Method and system for implementing a log parser in a log analysis system

Abstract

The present invention relates to methods and systems for implementing log parsers in log analysis systems. Systems, methods and computer program products are disclosed for implementing log analysis methods and systems that can configure, collect and analyze log records in an efficient manner. Improved methods have been described for automatically generating log parsers by analyzing the line content of logs. Furthermore, efficient methods have been described for extracting key-value content from log content.

Description

[0001] This application is a divisional application of the application patent application with application number 201680029404.0, application date 1 April 2016, and invention title "Method and system for implementing log parsers in a log analysis system". BACKGROUND

[0002] Many types of computing systems and applications generate large amounts of data related to or caused by the operation of that computing system or application. These large amounts of data are stored into collections such as log files / records, which can be reviewed at a later time period if there is a need to analyze the behavior or operation of the system or application.

[0003] Server administrators and application administrators can benefit by learning and analyzing the contents of system log records. However, collecting and analyzing these records can be a very challenging task. These challenges are for many reasons.

[0004] One significant problem involves the fact that many modern organizations have very large numbers of computing systems, each with a large number of applications running on those computing systems. Given the large number of disparate systems and applications running on these computing devices, it can be very difficult to configure, collect, and analyze log records in large systems. Moreover, some of these applications can actually run on and across multiple computing systems, making the task of coordinating log configuration and collection even more problematic.

[0005] Conventional log analysis tools provide basic capabilities for collecting and analyzing log records. However, when faced with the problem of large systems comprising large numbers of computing systems with large numbers of applications running on those systems, conventional systems do not scale efficiently. This is because conventional systems often work on a per-host basis, where whenever a new host is added or newly configured in a system, setup and configuration activities need to be performed, or even new log collection / configuration activities need to be performed for existing hosts. Given the large number of hosts in modern systems, this approach is very inefficient. Moreover, conventional approaches, especially on-premise solutions, also do not adequately allow for sharing of resources and analysis components. This results in a large and excessive amount of redundant processing and resource usage.

[0006] Conventional log analysis tools are also very inefficient when it comes to the construction of log parsers used by the log analysis tools. Log parsers are tools that understand how to parse entries within a log. Conventionally, log parsers must be manually constructed by people who must be both familiar with the exact format of the log files to be analyzed and skilled in the specific programming infrastructure that will be used to implement the parser.

[0007] One problem with the conventional approach of manually building a log parser is that this process requires a significant amount of time and resources from skilled technicians to build the parser. Furthermore, this approach requires excessive manual resources to maintain the parser as the log file format changes. Furthermore, this manual approach necessarily requires prior knowledge of the log file format.

[0008] Therefore, an improved method is needed to implement a log analysis system. It is also necessary to provide a more efficient way to implement a log parser for a log analysis system. Summary of the Invention

[0009] Some embodiments of the present invention solve the above problems by providing a method for automatically building a log parser. Instead of requiring a person to manually create the content of the log parser, the log content itself is used to build the parser.

[0010] According to some embodiments, a method, system, or computer-readable medium is provided that constructs a log parser by identifying a log to be analyzed, creating a mapping structure that maps the contents of the log to identified element types for one or more data portions within the log, selecting a data portion from the log, analyzing the data portion relative to the mapping structure to identify a variable portion and an immutable portion, assigning at least one of the variable portions to a least restrictive data type that encompasses the variability of values ​​detected in the at least one variable portion, and automatically generating a regular expression for the log parser. The regular expression may, in some embodiments, include an immutable portion and placeholders for the variable portion to implement the log parser, wherein at least two different placeholders are associated with different data types.

[0011] In some embodiments, the inventive method for identifying variable and immutable portions may be performed by identifying a line from a log to compare against a mapping structure, starting at the beginning of the line and moving forward until a mismatch is identified, finding the next common character, marking the middle range as variable, and looping until the end of the line is reached.

[0012] Within the mapping structure, the element type may include at least one of a string type, an integer type, an alphabetic character type, or a field rule type, wherein the field rule type corresponds to a sequence of elements defined by a rule.

[0013] Multiple rows are grouped together as a single entry for analysis against the mapping structure. In the alternative, the contents of multiple rows can be manipulated into a single row.

[0014] Delimiters for analysis ranges within a log can be identified by identifying common elements within two lines of the log, scoring the common elements, scoring the common elements by considering the position of the common elements in combination with one or more weighting factors, and selecting a common element as a delimiter based on the scoring results. In some cases, the weighting factors can include rules corresponding to combinations of multiple elements. Additionally, a sum or average can be calculated for the positions of the common elements.

[0015] Key and value fields can be extracted from a log by identifying a range for evaluating one or more key-value pairs by identifying a first key-value delimiter and iteratively identifying key-value pair delimiters within a line, and iteratively traversing the line to extract key fields from the left side of an instance of the key-value delimiter and value fields from the right side of an instance of the key-value delimiter. Preprocessing can be applied to the log to categorize the field and value portions of the log. In addition to or in lieu of preprocessing, post-processing can be applied to correct problematic assignments of content to key or value fields.

[0016] According to some embodiments, the log parser is employed in a log analysis system implemented as a cloud-based architecture and / or a SaaS (Software as a Service)-based architecture. The raw log data processed by the log analysis system can originate from any log generation source, such as a database management system (DBMS), a database application (DB App), middleware, an operating system, a hardware component, or any other log-generating application, component, or system. Log monitoring can be configured using a configuration mechanism that includes user-operable UI controls for selecting and configuring log collection configurations and target representations. The log collection configuration includes a set of information (e.g., log rules, log source information, and log type information) that identifies what data to collect (e.g., which log files), the location of the data to be collected (e.g., directory location), how to access the data (e.g., the format of the log and / or the specific fields to be obtained within the log), and / or when to collect the data (e.g., periodically). The target representation identifies a "target," which is a separate component that contains and / or generates logs. These targets are associated with specific components / hosts in the customer environment. The ability to configure log collection / monitoring by associating targets with log rules and / or log sources provides a unique advantage to the present invention. This is because users configuring log monitoring do not need to have a specific understanding of how logs for a given application are located or distributed across different hosts and components within the environment. Instead, users only need to select the specific target (e.g., application) to be monitored and then configure the specific parameters under which the log collection process will be performed.

[0017] Other additional objects, features and advantages of the present application are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0018] Various embodiments are described in the detailed description, drawings, and claims. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the figures. It should also be noted that the figures are merely meant to be illustrative and not limiting.

[0019] Figure 1A An example system that can be employed in some embodiments of the present application is shown.

[0020] Figure 1B A flowchart of a method that can be employed in some embodiments of the present application is shown.

[0021] Figure 2 A reporting UI is shown.

[0022] Figure 3A-3C A more detailed illustration of the internal structure of the log analysis system and the components within the customer environment that interact with the log analysis system is provided.

[0023] Figures 4A-4C A method of implementing log collection configuration is shown.

[0024] Figure 5 A flowchart of a method of implementing log collection configuration by associating log rules with targets is shown.

[0025] Figure 6 A flowchart of a method of implementing log collection configuration by associating log sources with targets is shown.

[0026] Figure 7 A flowchart of a method of implementing target-based configuration for log monitoring is shown.

[0027] Figure 8 A more detailed flowchart of a method of implementing target-based configuration for log monitoring in accordance with some embodiments of the present application is shown.

[0028] Figure 9 An example XML configuration content in accordance with some embodiments of the present application is shown.

[0029] Figure 10 Server-side information to be included in a configuration file to facilitate log parsing is shown.

[0030] Figure 11 A flowchart of one possible method of implementing this aspect of some embodiments of the present application is shown.

[0031] Figure 12 An architecture for implementing some embodiments of the inventive method of associating log analysis rules with variable locations is shown.

[0032] Figure 13 The extraction of additional data is shown to be inconsistent across all log entries.

[0033] Figure 14 Some example field definitions are shown.

[0034] Figure 15 A high-level flow chart of a method of implementing a log parser according to some embodiments of the present invention is shown.

[0035] Figure 16 A more detailed flowchart of a method of implementing a log parser according to some embodiments is shown.

[0036] Figures 17-1 to 17-21 Provides a diagram of the process of building a log parser.

[0037] Figure 18 A process flow for an embodiment that addresses non-standard row formats is shown.

[0038] Figure 19 Manipulation or classification of row contents is shown.

[0039] Figure 20 A flowchart of a method for efficiently identifying correct delimiter elements within a log content collection according to some embodiments of the present invention is shown.

[0040] Figures 21-1 to 21-5 The delimiter recognition process is shown.

[0041] Figure 22 Some example weights that may be applied to common elements in some applications of the present invention are shown.

[0042] Figure 23 A flowchart illustrating an example method of performing key-value extraction is shown.

[0043] Figures 24-1 to 24-12 The key-value extraction process is shown.

[0044] Figure 25-1 to Figure 25-2 and Figure 26-1 to Figure 26-2 An example row configuration is shown.

[0045] Figure 27 The architecture of an example computing system is shown with which the present invention may be implemented. DETAILED DESCRIPTION

[0046] As noted above, many types of computing systems and applications generate large amounts of data related to or resulting from the operation of the computing system or application. This large amount of data is then stored in collected locations such as log files / records that can be reviewed at a later time period if analysis of the behavior or operation of the system or application is desired.

[0047] Some embodiments of the present invention provide a method for automatically constructing a log parser. Instead of requiring a human to manually create the log parser content, the log content itself is used to construct the parser. Additional objects, features, and advantages of the present invention are described in the detailed description, drawings, and claims.

[0048] While the following description may describe the present invention with respect to "log" data, the scope of the present invention is not limited to the analysis of log data, and in fact the present invention can be applied to a wide range of data types. Therefore, the application of the present invention is not limited to log data, unless specifically claimed as such. Furthermore, the following description may also interchangeably refer to the processed data as "records" or "messages," without intending to limit the scope of the present invention to any particular format of data.

[0049] Log analysis system

[0050] This section of the disclosure provides a description of methods and systems for implementing large-scale log collection and analysis, which can be used in conjunction with a log parser constructed as described below.

[0051] Figure 1A An example system 100 for configuring, collecting, and analyzing log data according to some embodiments of the present invention is shown. System 100 includes a log analysis system 101, which in some embodiments is implemented as a cloud-based architecture and / or a SaaS (Software as a Service)-based architecture. This means that log analysis system 101 can provide log analysis functionality as a service on a hosted platform so that each customer who needs the service does not need to separately install and configure service components on the customer's own network. Log analysis system 101 can provide log analysis services to multiple individual customers and can be scaled to serve any number of customers.

[0052] Each customer network 104 can include any number of hosts 109. A host 109 is a computing platform within a customer network 104 that generates log data as one or more log files. Raw log data generated within a host 109 can originate from any log generating source. For example, raw log data can originate from a database management system (DBMS), a database application (DB App), middleware, an operating system, a hardware component, or any other log generating application, component, or system. One or more gateways 108 are provided in each customer network to communicate with the log analysis system 101.

[0053] The system 100 can include one or more users at one or more user stations 103 that use the system 100 to operate and interact with the log analysis system 101. A user station includes any type of computing station that can be used to operate or interface with the log analysis system 101 in the system 100. Examples of such user stations include, for example, a workstation, a personal computer, a mobile device, or a remote computing terminal. A user station includes a display device, such as a display monitor, for displaying a user interface to a user at the user station. The user station also includes one or more input devices, such as a mouse or keyboard, for the user to provide operational control over the activities of the system 100, such as for manipulating pointing objects in a graphical user interface to generate user input. In some embodiments, the user station 103 can (but need not) be located within a customer network 104.

[0054] The log analysis system 101 includes functionality accessible to a user at the user station 101, e.g., where the log analysis system 101 is implemented as a collection of engines, mechanisms, and / or modules (whether hardware, software, or a mix of hardware and software) to perform configuration, collection, and analysis of log data. A user interface (UI) mechanism generates a UI to display classification and analysis results, and to allow a user to interact with the log analysis system.

[0055] Figure 1B A flowchart of a method of using the system 100 to configure, collect, and analyze log data is shown. Figure 1B This discussion will refer to the components shown in the system 100 in Figure 1A

[0056] At 120, log monitoring is configured within the system. This can be performed, e.g., by a user / customer, to configure the type of log monitoring / data collection desired by the user / customer. Within the system 101, a configuration mechanism 129, including UI controls, is operable by a user to select and configure a log collection configuration 111 and a target representation 113 for a log collection configuration.

[0057] ​As discussed in more detail below, the log collection configuration 111 includes a collection of information (e.g., log rules, log source information, and log type information) that identifies what data to collect (e.g., which log files), the location of the data to be collected (e.g., directory location), how to access the data (e.g., the format of the log and / or specific fields within the log to be obtained), and / or when to collect the data (e.g., periodically). The log collection configuration 111 can include out-of-the-box rules included by the service provider. The log collection configuration 111 can also include customer-defined / customized rules.

[0058] Target representation 113 identifies a "target," which is an individual component within the customer environment that contains and / or generates logs. These targets are associated with specific components / hosts within the customer environment. An example target might be a specific database application that is associated with one or more logs and one or more hosts.

[0059] The ability of the current embodiment to configure log collection / monitoring by associating targets with log rules and / or log sources provides a unique advantage to the present invention. This is because a user configuring log monitoring does not need to have a specific understanding of how logs for a given application are located or distributed across different hosts and components within the environment. Instead, the user only needs to select the specific target (e.g., application) to be monitored and then configure the specific parameters under which the log collection process will be performed.

[0060] This solves a significant problem with conventional systems that require log monitoring to be configured on a per-host basis, requiring setup and configuration activities whenever a new host is added or newly configured in the system, or even requiring new log collection / configuration activities for existing hosts. Unlike conventional approaches, log analysis users can isolate the details of the exact hosts / components associated with the logs for a given target. This information can be encapsulated in underlying metadata maintained by system administrators, who understand the correspondence between applications, hosts, and components in the system.

[0061] The next action at 122 is to capture log data according to the user configuration. The association between log rules 111 and target representations is sent to the customer network 104 for processing. An agent of the log analysis system exists on each of the hosts 109 to collect data from the appropriate logs on the host 109.

[0062] In some embodiments, data masking can be performed on the captured data. Masking is performed at the time of collection, which protects customer data before it leaves the customer network. For example, various types of information in the collected log data, such as usernames and other personal information, can be sensitive enough to be masked before this information is sent to the server. Patterns are identified for such data, and before this data is collected for the server, it can be removed and / or replaced with proxy data. This allows the data to still be used for analysis purposes while hiding sensitive data. Some embodiments permanently remove sensitive data (e.g., changing all such data to "***" symbols), or replace it with data that is mapped so that the original data can be recovered.

[0063] At 124, the collected log data is delivered from the customer network 104 to the log analysis system 101. The plurality of hosts 109 in the customer network 104 provide the collected data to a reduced number of gateways in the one or more gateways 108, which then send the log data to the edge service 106 at the log analysis system 101. The edge service 106 receives the data collected from one or more customer networks and places the data into an inbound data store for further processing by the log processing pipeline 107.

[0064] At 126, the log processing pipeline 107 performs a series of data processing and analysis operations on the collected log data, which will be described in more detail below. At 128, the processed data is then stored into the data storage 110. The computer-readable storage 110 includes any combination of hardware and software that allows ready access to data located at the computer-readable storage 110. For example, the computer-readable storage 110 can be implemented as computer memory that is operatively managed by an operating system. The data in the computer-readable storage 110 can also be implemented as database objects, cloud objects, and / or files in a file system. In some embodiments, the processed data is stored within both a text / indexed data store 110a (e.g., as a SOLR cluster) and a raw / historical data store 110b (e.g., as a HDFS cluster).

[0065] At 130, reporting can be performed on the processed data using the reporting mechanism / UI 115. As shown in FIG. 2, the reporting UI 200 can include a log search facility 202, one or more control panels 204, and / or any suitable application 206 for analyzing / viewing the processed log data. Examples of such reporting components will be described in more detail below. Figure 2

[0066] ​Event management can be performed on the processed data at 132. One or more alert conditions can be configured within the log analysis system such that when an alert condition is detected, the event management mechanism 117 provides notification of the event / alert to a specified set of users.

[0067] At 134, the corrective action engine 119 can execute any necessary actions to be taken within the customer network 104. For example, a log entry may be received indicating that the database system is down. When such a log entry is identified, a possible automated corrective action is to attempt to restore the database system to service. The customer can create a corrective action script to address the situation. A trigger may be executed to run the script to perform the corrective action (e.g., the trigger causes an instruction to be sent to an agent on the customer network to run the script). In an alternative embodiment, the appropriate script for the situation is pushed down from the server to the customer network to be executed. Additionally, at 136, any other additional functions and / or actions may be taken as appropriate based at least on the processed data.

[0068] Figure 3A A more detailed diagram is provided of the internal structure of the log analysis system at the host environment 340 and the components within the client environment 342 that interact with the log analysis system. This architecture 300 is configured to provide a process for log monitoring that is capable of processing large amounts of log data digests.

[0069] In a client environment 342 within a single client host / server 344, an LA (Log Analysis) agent 333 retrieves log monitoring configuration data 332 (e.g., sniffer configuration or target-side configuration data) and invokes a log file 336 sniffer (also referred to herein as a "log collector") to collect log data from one or more log files 338. A daemon manager 334 can be used to interface with the log file sniffer 336. The log file sniffer 336 reads from one or more log files 338 on the host machine 344. The daemon manager 334 retrieves the log content and packages it so that it can be returned to the LA agent 333. It should be noted that the system can include any number of different types of sniffers, and the log sniffer 336 is merely an example of a single type of sniffer that can be used in the system. Therefore, other types of sniffers, such as sniffers for monitoring the registry, databases, Windows event logs, etc., can be employed within various embodiments of the present invention. Furthermore, in some embodiments, the log sniffer is configured to process aggregated / compressed files, such as Zip files.

[0070] LA agent 333 sends the collected log data to gateway agent 330. Gateway agent 330 packages the log data collected from multiple customer hosts / servers, essentially acting as an aggregator for log content from multiple hosts. The packaged content is then sent from gateway agent 330 to edge service 306. Edge service 306 receives a large amount of data from multiple gateway agents 330 from any number of different customer environments 342.

[0071] Given the potentially large amount of data that can be received at the edge service 306, the data is immediately stored in an inbound data storage device 304 ("platform inbound repository"). This acts as a queue for the log processing pipeline 308. A data structure is provided to manage items to be processed within the inbound data repository. In some embodiments, a messaging platform 302 (e.g., implemented using a Kafka product) can be used to track pending items within the queue. Within the log processing pipeline 308, a queue consumer 310 identifies the next item to be processed in the queue, which is then retrieved from the platform inbound repository. Queue consumers 310 include any entity capable of processing work that leaves a queue within the system, such as a process, thread, node, or task.

[0072] The retrieved log data undergoes a "parsing" phase 312 where the log entries are parsed and broken down into specific fields. As discussed in more detail below, the "log type" configured for the log specifies how the log entries are broken down into the desired fields.

[0073] In the "normalize" stage 314, the identified fields are normalized. For example, a "time" field may be represented in any number of different ways in different logs. The time field may be normalized to a single recognizable format (e.g., UTC format). As another example, the word "error" may be represented in different ways in different systems (e.g., all uppercase "ERROR", all lowercase "error", capitalized "Error", or the abbreviation "err"). This situation may require normalizing the different word forms / types to a single format (e.g., all lowercase, non-abbreviated term "error").

[0074] The "transformation" stage 316 can be used to synthesize new content from the log data. As an example, and as will be discussed in more detail below, "tags" can be added to the log data to provide additional information about the log entry. As another example, field extraction can be performed to extract additional fields from existing log entry fields.

[0075] The "Condition Evaluation" stage 318 is used to evaluate the log data for specified conditions. This stage can be performed to identify patterns within the log data and to create / identify alert conditions within the logs. Any type of notification can be performed at this stage, including, for example, an email / text message / call to an administrator / customer or an alert to another system or mechanism.

[0076] The log writer 320 then writes the processed log data to one or more data stores 324. In some embodiments, the processed data is stored in both a text / indexed data store (e.g., as a SOLR cluster) and a raw and / or historical data store (e.g., as an HDFS cluster). The log writer can also send the log data to another processing stage 322 and / or a downstream processing engine.

[0077] like Figure 3B As shown in , some embodiments provide a side loading mechanism 350 to collect log data without going through the agent 333 on the client side. In this approach, a user logs into the server to select one or more files on the local system. The system will load the file at the server and sniff the file (e.g., by having the user provide a log type, trying possible log types, rolling through different log types, or by making an educated "guess" about the log type). The sniffing results are then passed to the edge service and processed as previously described. Figure 3C In an embodiment, only the side loading mechanism 350 exists to collect log files - where a proxy / sniffer entity is not installed and / or required on the client server 344.

[0078] Figure 4A-4B A method for implementing log collection configuration is shown. This method allows for very large-scale configuration of how to monitor a log file with one or more log entries. In some embodiments, a log entry corresponds to a single logical line from the log file. In actual log files, a single entry can occupy multiple lines because carriage returns are part of the log entry content. This entire content is considered a single "entry". Each entry begins with "#### <date”开头,并且可以在文件中占用单个物理行或者由回车分隔的多个行。

[0079] In this model, "Log Type" 406 defines how the system reads log files and how to break log files into their components. In some embodiments, log files contain several basic fields. The basic fields present may vary for different types of logs. A "Basic Parser" can be used to break log entries into specified fields. A basic parser can also perform transformations. For example, a date field can be converted to a normalized format and the time adjusted to UTC so that data from multiple locations can be mixed together.

[0080] "Log Source" 404 defines where the log files are located and how to read them. In some embodiments, a log source is a named definition that contains a list of log files described using a pattern and the parser required to parse the file. For example, a source may be "SSH Log Files". This source may list each log file related to SSH separately, or may use wildcards (e.g., " / var / log / ssh*") to describe the log files. For each pattern, a basic parser may be selected (e.g., by the user) to parse basic fields from the file. This approach can be used to ensure that for a single pattern, all files conform to the same basic parsing structure. For a source, multiple log types can be selected and priority is given to those possible types. For example, types A, B, and C can be identified, with the analysis working through each of these types to determine whether the source matches one of these identified types. Therefore, for each pattern, the user can select multiple basic parsers. In some embodiments, the same source can be matched against multiple types and can be analyzed using multiple types.

[0081] "Log rules" 402 define a collection of sources along with conditions and actions to be triggered during ongoing monitoring. "Targets" 408 identify individual components in the IT environment that contain logs. In some embodiments, associating a rule to a target initiates the monitoring process.

[0082] exist Figure 4A In an embodiment, one or more log rules are associated with one or more targets. Figure 4B In an alternative embodiment, one or more log sources can be associated with one or more targets to create an instance of the target. Figure 4C In the embodiment of

[0065] , log rules are not even provided as a method for creating associations—only associations of log sources to targets are provided to create target instances. Each of these methods will be described in more detail below.

[0083] Figure 5A flow chart illustrating a method for implementing log collection configuration by associating log rules with targets is shown. At 502, one or more log rules are created. These rules are processed by a rule engine within the log processing system to implement rule-based processing for a given target. Thus, a rule will include specific logic for processing the given target to which it is associated.

[0084] In some embodiments, rules can be used to specify a target type, which identifies the type of target that the rule is intended to address. Rules can be specified for a single target type or for multiple target types. For example, when monitoring log files for a database instance, the target type can be set to "database instance" so that activity in the log is reported for the appropriate target type. In some embodiments, even though a rule can be configured for "file" as the log type, the target type can still be any managed target type, such as database.

[0085] A rule can specify a source type, which identifies the type of log file the rule is intended to handle. For example, a rule can specify that the log file type will be: (i) File: OS-level log file; (ii) Database Table: a table in the database that stores log content; (iii) Windows Event Log: read events from Windows Event Log as log content.

[0086] Target attribute filters can be specified in rules to filter targets to specify conditions for which the rule applies, such as a specific operating system (OS), target version, and / or target platform. For example, a user can create a rule that only targets a given OS on a given platform (e.g., only Linux OEL5 on x86_64 hardware).

[0087] In some embodiments, when creating a rule, the rule may also include: (a) a name of the rule; (b) a severity level that indicates how important the consequences of the rule are if the rule causes an event to be generated; (c) a description of the rule; and / or (d) a textual justification for why such monitoring is being performed.

[0088] In some embodiments, one or more conditions may be established for which a rule will "trigger". Multiple conditions may be specified, each of which may be combined with other conditions using Boolean operators. For example, a set of conditions that are OR'ed with other conditions means that if any of the conditions matches an entry in the log file being evaluated, then that entry triggers this rule. When conditions are AND'ed together, all clauses of the condition must be satisfied in order for the condition to trigger an entry in the log file. The specified action will then be taken in response to the entry being matched. The following is an example conditional clause that includes a regular expression: "MESSAGE contains "START: telnet pid=[0-9]*from=[.]*"", where if the message matches the regular expression, then this condition triggers the rule.

[0089] The "operator" in the condition is how the comparison is performed. The following are some example operators that may be employed in some embodiments of the present invention: (a) <, >, >=, <=: compares a value greater than or less than (or equal to) a set value; (b) Contains: pattern matching with the ability to include regular expression clauses, where implicit wildcards can be placed at the beginning and end unless the user uses the ^ and $ regular expression symbols to specify the beginning or end of a string; (c) In: a list of possible values; (d) Is: an exact string match (no regular expression capability); (e) Is Not; (f) Does Not Contain; (g) Not In: a list of values ​​that do not match.

[0090] Actions can be specified to identify what to do when a match is found for a given condition on a selected source. For example, one possible action is to capture the full log entry as an observation when a rule's conditions are matched. This approach allows the system / user to monitor logs from any source and, when a single entry matching the rule's conditions is seen, save the full entry and store it as an observation in the repository.

[0091] The observations are stored for later viewing through the log observation UI or other reporting features. Another possible action is to create an event entry for each matching condition. This method triggers an event when a log entry is deemed to match the specified conditions. In some embodiments, the event will be created directly at the agent. The source definition will define any special fields (if any) that may be needed to capture the event. An additional option for this action is to bundle duplicate log entries at the agent and report the event only once within a user-specified time range. Matching conditions can be used to help identify the existence of duplicate entries. Another example action is to create metrics for the rule to capture each occurrence of the matching conditions. In this method, a metrics subsystem is used to create new metrics for this rule. Thereafter, when there is a log entry that matches the conditions of the rule, a certain number of fields are captured as metric data and uploaded as part of this metric. These fields can be selected to include information such as "key" fields (such as target, time, source, etc.).

[0092] At 504, one or more targets are identified in the system. Targets are individual components within the customer environment that contain logs. These targets are associated with specific components / hosts within the customer environment. Example targets include hosts, database applications, middleware applications, and / or other software applications associated with one or more logs or hosts. Further details regarding the method for specifying targets are described below.

[0093] At 506, an association is made between the target and the rule. Metadata can be maintained in the system to track the association between a given target and a given rule. A user interface can be provided that allows the user to view what targets the selected rule is associated with and / or add more associations, where association is the way to make a rule active by associating it with a real target.

[0094] Thereafter, log collection and processing are performed based at least in part on the association between the rules and the targets at 508. As discussed in more detail below, target-based configuration may involve various types of configuration data created at both the server side and the target side to implement log collection and log processing.

[0095] The current embodiment provides a unique advantage by enabling the configuration of log collection / monitoring by associating targets with log rules. This is because users configuring log monitoring do not need to have a specific understanding of how logs for a given application are located or distributed across different hosts and components within the environment. Instead, users simply select the specific target (e.g., application) to monitor and then configure the rules by which the log collection process will be performed.

[0096] This solves a significant problem with conventional systems that require log monitoring to be configured on a per-host basis, requiring setup and configuration activities whenever a new host is added or newly configured in the system, or even requiring new log collection / configuration activities for existing hosts. Unlike conventional approaches, log analysis users can isolate the details of the exact hosts / components associated with the logs for a given target. This information can be encapsulated in underlying metadata maintained by system administrators, who understand the correspondence between applications, hosts, and components in the system.

[0097] Instead of or in addition to rules, you can configure log processing by associating log sources with targets. Figure 6 A flow chart of a method for implementing log collection configuration by associating log sources with targets is shown. At 602, one or more log sources are created. A log source defines where log files are located and how they are read. A log source can define a source type that indicates how the source content is collected. The following are example source types: (a) File - identifies a readable file from the OS level that can be accessed using regular OS-level file operations; (b) Database Table - a table that stores log entries (e.g., a database audit table); (c) Windows Event System - an API that provides access to event records. One or more source names can be defined for a log source. In addition, a log source can be associated with a description of the source. Note that log sources can also be used when creating log monitoring rules (as described above).

[0098] Log sources can also be associated with file patterns and / or pathname expressions. For example, " / var / log / messages*" is an example of a file pattern (which may actually be associated with multiple files). Regarding file patterns, one reason they are used in current log analysis systems is because it is possible that the exact location of the logs to be monitored will change. There are times when a system will expect logs to be located in a specific place, such as in a specific directory. When a system is processing a large number of streaming logs, it may not be clear to the system in which directory the logs are expected to be located. This prevents systems that rely on static log file locations from functioning correctly. Therefore, file patterns are useful for addressing these potentially changing log locations.

[0099] In some embodiments, a log source is created by specifying a source name and description for the log source. The definition of a log source can include included file name patterns and excluded file name patterns. A file name pattern is a pattern corresponding to the files (or directories) to be included for the log source. An excluded file name pattern corresponds to a pattern of files (or directories) that are explicitly excluded from the log source, for example, this is useful where an included file name pattern identifies a directory with many files and some of these files (such as pseudo files or non-log files) are excluded using an excluded file name pattern. For each pattern, the system captures the pattern string, description, and basic parser (log type) that will be used to parse the file. The basic parser can define the basic structure of the file, for example, how to parse data, host names, and messages from the file.

[0100] The definition of a log source can also specify whether the source contains security log content. This allows the source creator to specify a special role that users must have in order to view any log data that may be captured. This log data can include security-related content that is not viewable by any target owner.

[0101] As noted above, log rules can reference log sources, and vice versa. In some embodiments, system metadata tracks these associations to maintain a count of rules currently using a source. This helps understand the impact if a source and / or rule is changed or deleted.

[0102] At 604, one or more targets are identified. As noted above, a target is a component within the environment that contains, corresponds to, and / or creates logs or other data to be processed, where a target is associated with a specific component / host in the customer environment. Example targets include hosts, database applications, middleware applications, and / or other software applications associated with one or more logs or hosts.

[0103] At 606, an association is made between the target and the source. Metadata can be maintained in the system to track the association between a given target and a given source. A user interface can be provided that allows the user to view what targets the selected source is associated with and / or add more associations.

[0104] At 608, the association of a target to a source creates a specific instance of the log source. For example, consider a log source that typically specifies that a given file is located at a given directory location (e.g., c: / log_directory / log_file). It may be the case that any number of servers within the customer environment (Server A, Server B, Server C, Server D) may have copies of that file (log_file) in that directory (c: / logdirectory). However, by associating a specific target (e.g., Server A) to the log source, this creates an instance of the log source so that the new instance is specific to the log file in the specified directory on the specific target (e.g., begins monitoring c: / log_directory / log_file specifically on Server A).

[0105] Thereafter, log collection and processing are performed based at least in part on the association between the rules and the log sources at 610. As discussed in more detail below, target-based configuration can involve various types of configuration data created at both the server side and the target side to implement log collection and processing activities.

[0106] There are many benefits to using this type of model for configuring log collection. One benefit is that log types, sources, and rules can be easily reused as needed. Furthermore, this approach avoids the need for extensive reconfiguration by enabling sharing at multiple levels. Furthermore, users can create custom rules that use sources and log types defined by others or provided with the product. This approach also makes it easy to build on shared knowledge.

[0107] Associating rules / sources with targets provides knowledge of where log collection is physically performed via the agent. This means users don't need to know anything about the location of the targets. Furthermore, this facilitates large-scale association of rules / sources with targets. In some embodiments, rules / sources can be automatically associated with all targets based on configuration. As noted above, service providers can provide out-of-the-box configurations. Furthermore, users can create their own configurations, including extending the provided out-of-the-box configurations. This allows for customization without requiring users to build their own content.

[0108] Figure 7 A flow chart illustrating a method for implementing target-based configuration for log monitoring is provided. This process generates the creation, deployment, and / or updating of configuration profiles for log monitoring. In some embodiments, the configuration profiles are implemented as configuration files used by the log monitoring system to manage and implement the log monitoring process.

[0109] At 700, target-based processing is initiated. An example method for initiating target-based processing includes, for example, installing a log analysis agent to a specific log collection location. Target-based processing involves associating one or more targets with one or more log sources and / or rules.

[0110] At 702, configuration data is generated for target-based processing. In some embodiments, the target-based configuration data is implemented as a configuration XML file, but other formats can also be used to implement the configuration data. The target-based configuration data can be created at the master site (e.g., to create a master version 704), and then the specific version is transferred to the server side and the target side.

[0111] Target-side profile 708 may include configuration details related to log collection. This includes, for example, information about log source details and target details. Server-side profile 706 may include configuration details related to server-side log processing. This includes, for example, information about parser details.

[0112] In some embodiments, a database at the server maintains a master version and a target version of the configuration profile. As noted above, the target version includes configuration details relevant to the log collection effort and is passed to the client environment for use by agents in the client environment to collect appropriate log data from the client environment. The master version includes the full set of configuration details required at the server and becomes the "server-side" profile when selected and used for processing at the server. This can occur, for example, when log data collected at a target is passed to the server, where the sending of the log data includes an identifier (e.g., Figure 9 The configuration version or "CV" number 903 is shown in the example target side profile of FIG. 903). When this data is received at the server, the identifier is used to determine the corresponding major version of the profile having the same identifier number (e.g., Figure 10 ). This major version is then used as the server-side profile to process the received log data. Thus, in this embodiment, major version 704 and server-side profile 706 are identical, but have different labels depending on whether the profile is currently being used to process log data. In alternative embodiments, for example, where the profile is used on multiple servers with different configuration details, the major version may be different from the server version.

[0113] At 710, the configuration data is then distributed to the appropriate location within the log processing system. In some embodiments, the target side data 708 is used as Figure 3A The sniffer configuration file 332 shown in FIG is distributed to the client system. Figure 1A The log configuration file 111 shown in FIG is "distributed," where the distribution does not actually require the material to be distributed across the network, but merely indicates that the material is obtained from another component within the server (eg, as needed).

[0114] Thereafter, at 712, a log collection process is performed at the target using the target-side configuration profile. Additionally, at 714, a server-side log process is performed using the server-side configuration profile.

[0115] Figure 8 A more detailed flow chart of a method for implementing target-based configuration for log monitoring according to some embodiments of the present invention is shown. At 802, one or more work items are created in the system for processing target associations. For example, this type of work can be created when a log analysis agent is installed on a target, where the identification of such installation causes a work item to be created for target-based configuration data. A list of target types having at least one automatic association rule (e.g., from an associated database) is identified. A list of targets that need to be associated with the automatically enabled rules is generated. These steps are equivalent to a producer entity / process placing an association task into a queue (e.g., a database table), which is then processed by one or more consumer entities / processes.

[0116] One or more consumer / worker entities may wake up periodically to process work items. For example, a worker entity (e.g., a thread or process) wakes up (e.g., every 10 seconds) to check if there are any pending associated tasks. A collection of one or more workers will iterate through the tasks to process the work in the queue.

[0117] At 804, one of the workers identifies the association task to be processed. At 806, the association request is processed by accessing the information collected for the rule, source, parser, field, and / or target. This action identifies the target being addressed, finds the target, and then looks up the details of the log source and / or log rule that has been associated with the target.

[0118] At 808, the worker then generates configuration content for the specific associated task it is processing. In some embodiments, the configuration content is implemented as XML content. This action creates both target-side details and server-side details for the configuration profile. For the server-side, this action will create configuration data for the server to process the collected log data. For example, resolver details in XML format are created for the server-side profile to be used for the log data expected to be received. For the target-side, this action will create configuration data for log collection from the target. For example, as discussed below, variable pathnames (e.g., with variables instead of absolute pathnames) can be specified for a given log source to identify the directory containing the log files to be monitored. At step 808, these variable pathnames can be replaced with actual pathnames and inserted into the target-side profile.

[0119] At 810, it is determined whether there are any additional associated tasks to process. If there are additional tasks on the queue, then the process returns to 804 to select another task to process. If not, then at 812, the configuration profile is finalized.

[0120] It is noted that the same configuration / XML file can be used to address multiple associations. For example, if multiple targets are located on the same host, a single configuration file can be generated for all of the targets on the host. In this case, step 808 described above appends the XML content to the same XML file for multiple iterations through the processing loop.

[0121] Updates can occur in a similar manner. When a change occurs that requires an update to the profile, one or more new associated tasks can be placed on the queue and addressed as described above. In addition, disassociation can also occur, for example, in the case of a log analysis agent being uninstalled. In this case, the configuration file can be deleted. When a target is deleted, a message can be broadcast to all listeners through the target model service that can be consumed to delete the corresponding association and update the XML content.

[0122] Figure 9Example XML configuration content 900 is shown according to some embodiments of the present invention. This is an example of target-side content that can be placed on a host that holds a target. This XML configuration content 900 defines rules for collecting Linux system message logs with the file pattern " / var / log / messages*" on host XYZ.us.oracle.com. Section 902 identifies the base parser for the association being addressed. Section 903 provides an identifier for the version number ("configuration version" or "CV") of content 900, which is used to match corresponding server-side material with the same version number. Section 904 identifies the ID of the log rule. Section 906 identifies the specific target. Section 908 identifies the target type. Section 910 identifies the source type. Section 912 identifies the parser ID for the source. The logs will be parsed based on some defined parser. These configuration files reside on the sniffer, and the log collection process collects logs based on the defined log sources.

[0123] In the server-side log processor, additional information can be included in the configuration file to facilitate log analysis, e.g. Figure 10 1000. The server-side content of the content 1000 is shown in section 1000. FieldDef section 1001 indicates the data type of the service. Log Source section 1002 indicates that the log has the "os_file" type. BaseParse section 1004 defines how to parse the log entry based on the regular expression defined in section 1006. Section 1003 provides an identifier for the version number of the content 1000, which is used to match the corresponding target-side material with the same version number.

[0124] In addition to the automatic association described above, it is also possible to perform a manual target-source association. For example, a user interface can be provided to perform a manual association. This also results in the above actions being performed, but triggered by a manual action.

[0125] Resynchronization of target-source associations can be performed. To explain, consider that when a log analysis agent is installed, monitored targets connected through the agent can be associated with certain predefined log sources. Similarly, when the agent is uninstalled, these associations can be deleted from the appropriate database tables. Furthermore, when a target is added for monitoring by the agent, the target can be associated with certain predefined log sources for that target type, and when the target is deleted from the agent, this association can be deleted from the database tables.

[0126] Over time, these associations can become out of sync for various reasons. For example, when installing a log analysis agent, an automatic association might occur due to some network issue that causes the configuration data to be lost during its transmission. Additionally, when adding or removing targets, events might not be processed correctly, so updates to the configuration XML file don't occur properly.

[0127] In order to handle these situations and maintain the consistency of the association between the target and their corresponding log source, a web service is provided in some embodiments to periodically synchronize the association. In at least one embodiment, only automatic associations are synchronized, while manual associations manually customized by the user are not synchronized.

[0128] Association can be performed for specific log analysis agents. Incremental analysis can be performed between targets in the data model data repository and targets in the log analysis data repository to achieve this action. Processing can occur in the following situations: (a) for targets that are in the data model data repository but not in the log analysis data repository, associations are added for these targets; (b) for targets that are not in the data model data repository but in the log analysis data repository, associations for these targets are deleted; (c) for targets in the data model data repository and the log analysis data repository, the same associations are retained for these targets in the case of user customization. A potential problem with adding associations is related to the following situation: the user may have deleted all associations for a specific target, so there is no entry in the log analysis data repository, but there is an entry in the data model data repository. The problem is that when applying the above method, unwanted automatic associations may be introduced again after the synchronization operation. To avoid this situation, the system can log user actions to identify potential problems.

[0129] Additionally, associations can be synchronized for a specified tenant. When this action is performed, incremental analysis can be performed between the agent for the data model data repository and the agent for the log analytics data repository. Processing can occur by: (a) adding associations for agents that are in the data model data repository but not in the log analytics data repository; (b) removing associations for agents that are not in the data model data repository but in the log analytics data repository; and (c) performing the same incremental analysis and synchronization as described above for agents in both the data model data repository and the log analytics data repository.

[0130] A synchronization can be performed for all tenants' associations. When this action is performed, the agent-level synchronization described for each tenant should be performed.

[0131] Turning the attention of this document to file patterns, one reason for using file patterns in a log analysis system is because the exact location of the logs that are to be monitored can vary. At most times, the system will expect the logs to be in a particular location, in a specific directory. When the system is handling a large number of streaming logs, it can not be clear in which directory these logs are expected to be located. This prevents systems that rely on static log file locations from operating correctly.

[0132] In some embodiments, the inventive method can associate log analysis rules to variable locations. One approach is to use metadata that replaces the variable part that corresponds to the location of the log file. A path expression is used to represent the path name of the log file, where the path expression includes a fixed part and a variable part, and different values are realized for the variable part. The placeholder for the location is eventually replaced with the actual location in the directory path.

[0133] Some embodiments provide "parameters" that are flexible fields (e.g., text fields) that a user can use when including file name patterns or excluding file name patterns. Parameters can be implemented by including the parameter name in curly braces { and}. A user-defined default value is provided in this source. The user can then provide parameter overrides on a per-target basis when associating the log monitoring rules that use this source to targets. Overriding is especially useful, for example, for changes from out-of-the-box (OOTB) content (e.g., to override rules, definitions, etc. without actually changing the OOTB content). This is implemented, for example, by implementing a mapping / annotation table that includes user overrides and indicates overrides to OOTB content.

[0134] This is very helpful because, in a log source, a path can be defined for the log files that are to be monitored. In some cases, the path is fixed, such as in a Linux syslog file, the path is " / var / log / messages*". However, in other cases, it can be desired to monitor database alert logs, where each database target will be installed in a completely different path, and the path to the alert log can be different. For example, the alert log for one database is located at the location: / xxx / db / yyyy / oracle / diag / rdbms / set2 / set2 / alert / log*.xml". The underlined part can be different for each database target. However, each target has the concept of target attributes. Included in these attributes is metadata that can be used to fill in the variable part in the path. In the current embodiment, this path can be alternatively represented as:

[0135] "{DIAGNOSTIC_DEST} / diag / rdbms / {SID} / {SID} / alert / log*.xml"

[0136] When this source is used in a rule and this rule is associated with a target, the system replaces the parameters "DIAGNOSTIC_DEST" and "SID" with the parameters known for that target. This allows the system to associate a single rule and source with thousands of targets at once.

[0137] As another example, a user might want to monitor the pattern: " / xxx / oracle / log / *". In this case, " / xxx / oracle" is a variable path that depends on the host. The pattern could instead be written as: "{INSTALL_DIR} / log / *". For this source, the user could provide a default value ( / xxx / oracle) for the INSTALL_DIR parameter. Later, when a rule is associated with a target, the user could provide a target override value of " / xxx / oracle" for this parameter on this target without creating a new source or rule.

[0138] For system-defined fixed parameters, there may be situations where the user wishes to reference a built-in parameter (e.g., ORACLE_HOME). In this case, the system will replace that variable with the ORACLE_HOME known for the selected target. The pattern can be written as: "{ORACLE_HOME} / log / *". This path will be automatically understood by the agent, where ORACLE_HOME is a special built-in parameter that does not require the user to set a default value. The system can be provided with a list of fixed parameters that the integrator / user can choose to use.

[0139] Figure 11 A flow chart illustrating one possible method for implementing this aspect of some embodiments of the present invention is shown. At 1102, location content is identified for which variable location processing is desired. This situation may exist, for example, when the system is processing a large number of streaming logs from a potentially large and / or indeterminate number of directory locations. The log data may be located at a target location addressed using a pathname that varies for different database targets.

[0140] At 1104, a path having a fixed portion and a variable portion is specified for a target location. The variable portion may be represented by one or more parameters. During log processing, at 1106, the one or more parameters are replaced with values ​​corresponding to one or more target log files, wherein a single rule for implementing log monitoring is associated with multiple different targets to be monitored.

[0141] This approach is quite advantageous over approaches where each log is in a different directory that cannot be known in advance and a separate forwarder mechanism would have to be set up for each path. In contrast, this approach can be used to set up one rule for a very large number of paths.

[0142] In some embodiments, configuration information from the log analysis system can be coupled to this method to configure and set rules for identifying log file assignments. Some examples of configuration information that can be used include, for example, how databases are connected, how components are connected, which data center to use, etc.

[0143] Some embodiments specify how to map sources to targets based on their relationships. For example, a defined source Source1 can be assigned to all related targets belonging to a system. Any association type and / or rule can be used in this embodiment, for example, where a common set of association types is used to provide configuration information useful for determining rules for log locations. Such association types may include, for example, "contains", "application_contains", "app_composite_contains", "authenticated_by", "composite_contains(abstract)", "cluster_contains", "connects_through", "contains(abstract)", "depends_on(abstract)", "deployed_on", "exposes", "hosted_by", "installed_at", "managed_by", "monitored_by", "provided_by", "runs_on(abstract)", "stores_on", "stores_on_db", and "uses(abstract)".

[0144] It should be noted that the target relationship information / model can also be used in other ways. For example, the target model can also be used to help correlate log entry lookup results to aid in root cause analysis. As another example, the host model can be used to compare all hosts in a system. For example, if there are multiple databases in a first system, this feature can be used to view logs across these systems together, and isolated from the database for a second system.

[0145] Figure 12The architecture of some embodiments of the inventive method for associating log analysis rules with variable locations is shown. Here, the log analysis engine 1202 operates by accessing a log collection configuration file 1211. The log collection configuration file 1211 is implemented to represent a path in which the target location can have both a fixed portion and a variable portion. The variable portion can be represented by one or more location parameters. In this example, different locations can exist for logs 1202a, 1201b, and 1201c. By replacing the variable portion, the specific location of the log of interest can be selected by the log analysis engine 1202 and processed to generate analysis results 1213.

[0146] Here, reference material 1210 can be accessed to identify the correct replacement for the variable portion of the path to the target location. Any suitable type of reference material can be implemented. As mentioned above, the defined source Source1 can be assigned to all relevant targets belonging to a certain system, and / or association types and / or rules can also be used. In addition, target relationship information / models and reference materials can be used.

[0147] Thus, embodiments of the present invention provide improved functionality for performing target-based log monitoring. Two possible use cases for this functionality include log monitoring and self-organizing log browsing. Log monitoring involves, for example, situations where there is ongoing monitoring and capture of logs. Some embodiments of log monitoring involve some or all of the following: (a) monitoring any log for any target and capturing important entries from the log; (b) creating events based on some log entries; (c) identifying the existence of log entries that can affect compliance scores; (d) performing user-defined monitoring as well as integrator-defined monitoring; (e) capturing log entries that are not events to enable analysis on a subset of all logs; (f) use cases such as intrusion detection, potential security risk detection, and problem detection; (g) enabling long-term persistent storage of log content; (h) searching log content; (i) customizable search-based views; and (j) log anomaly detection and scoring.

[0148] Ad hoc log browsing addresses situations where, for example, there's no ongoing log monitoring. In this approach, users can browse live logs on the host without having to collect them and send them up to the SaaS server. The model for configuring what to monitor is similar to the one described previously. The difference lies in the fact that users can select rules, sources, and some filters from the UI. The search is then sent down to the agent to obtain matching log files and bring them back, storing them in temporary storage on the server. The user can continue to narrow the search on this result set. If the user adds another target, rule, or expands the time range, the system returns to the agent to obtain only the incremental content, rather than retrieving the entire content. Thus, users can obtain the same benefits of log analysis without having to configure ongoing log monitoring. This feature can achieve very low latency because the system only needs to return to the agent to obtain more data when the search is expanded. All searches that narrow the current result set are compared to previously acquired, cached data from the agent.

[0149] Embodiments of the present invention can be used to store log data in a long-term centralized location within a raw / historical data repository. For example, a target owner in a company's IT department can monitor incoming issues for all targets they are responsible for. This can include thousands of targets (hosts, databases, middleware, and applications) managed by the company's SaaS log analysis system. Many log entries (e.g., hundreds of GB of entries) can be generated daily. For compliance reasons, these logs may need to be permanently stored, and based on these logs, data center managers may want to obtain a long-term overview (big picture), and IT administrators may want to search through them to identify possible causes of specific issues. In this scenario, very large amounts of logs can be stored in a centralized storage device, where users can search the logs and view log trends with acceptable performance. In some embodiments, log data can be stored in an offline repository. This can be used, for example, when data is kept online for a period of time and then transferred offline. This is particularly useful when there are different pricing tiers for different types of storage devices (e.g., offline storage devices are less expensive) and users can choose where to store their data. In this approach, data can be kept in offline storage devices and can be brought back online at a later point in time.

[0150] Logs can be searched to analyze possible causes of problems. For example, when a particular problem occurs for a target, the target owner can analyze logs from various sources to find out the cause of the problem. In particular, time-related logs from different components of the same application or from different but related applications can be reported in a time-interleaved format in an integrated view to help the target owner find possible causes of the problem. The target owner can perform some ad hoc searches to find the same or similar log entries over time and jump to the log entry of interest, and then drill down to the detailed message and browse other logs generated before / after the point of interest.

[0151] In some embodiments, restrictions can be applied so that users can only access logs for which they have been given access permissions. Users of different levels can be associated with access to different sets of logs. Various roles can be associated with permissions to access certain logs.

[0152] Some embodiments may be employed to view long-term log distribution, trends, and correlations.Since many logs are generated over a long period of time by many different targets and log sources, data center managers may want to view long-term log distribution and patterns.

[0153] Some embodiments can be used to search logs to identify the cause of an application outage. Consider a scenario where the target owner or IT administrator of a web application receives some notification that some customers using the application report that they are unable to complete their online transactions and that the confirmation page cannot be displayed after clicking the submit button. Using embodiments of the present invention, the IT administrator can search the logs generated by the application using the username as a keyword and within the time range of the problem report. Some application anomaly may be found in the logs indicating that some database error occurred when the application tried to submit a transaction. By adding databases and their corresponding hosting servers for search via target associations and their availability-related log sources, the IT administrator can browse the logs around the time of the application anomaly to find some database errors, which are related to, for example, some hosting server partial disk failure and a large number of submitted transactions.

[0154] Some embodiments can be used to view long-term log distribution, trends, and correlations by tag. A data center manager can define tags for logs collected in the data center, such as security logs for production databases, security logs for development servers, logs for test servers, noise logs, and so on. The data manager may be interested in understanding, for example, the distribution of logs by these tags over the past six months, their daily incoming rates during the past month, and whether there is any correlation between security log entries for production databases and changes in their compliance scores during a given time period.

[0155] Some embodiments allow log data to be stored as metrics. In certain embodiments, the system will store several log fields as key fields. Key fields will include (but may not be limited to): Time, Target, Rule, Source, and Log File. The system may also create a hash or GUID to distinguish between possible log entries with the same time and all other key fields. When a rule using this metric action for a log entry is associated with the first target, a metric extension is created and deployed. This metric extension will be named similarly to the rule to make it easier for users to reference it.

[0156] In some embodiments, a log monitoring rule has a possible action for creating an event when a log entry matches the conditions of the rule. In addition, the user will be able to indicate that this event should also trigger a compliance violation, which will affect the compliance score for the compliance standards and frameworks.

[0157] As noted above, one possible use case is to provide a log browser, for example, where browsing is used to browse live logs on a host, rather than collecting the logs and sending them to a SaaS server. A user can select rules, sources, and certain filters from the UI, and then searches are sent down to the agent to obtain matching log files and bring them back so that they are stored in temporary storage in the server. One use case for this feature is to allow users to browse short periods of time in log files across multiple targets in the system to try to discover the source of a problem, especially where there are rich topology maps and dependency maps for the customer environment. This content can be used to help find related elements and show logs together. This allows the user to view logs for, for example, all targets related to a given system and see what happened across all targets in chronological order. In many cases, when there is a target failure, it may be a dependent target that is experiencing the problem, rather than the target that is failing.

[0158] The user can choose to start a new log browsing session in the context of the system / group / individual target. If entered from the target home page, the target home page context will be retained. This means that the shell of the page still belongs to the target home page, and only the content panel will contain the browsing UI functionality. This means that the browsing UI can be implemented as modular so that it can be dynamically inserted into other pages. In some embodiments, multiple lines of content can be provided for each entry to show additional details for each line. This is done one line at a time, or the user can decide to do this for all lines. Sorting of the parsed fields can be provided, but in addition, sorting can be used to view additional details for each line (including the original log entry).

[0159] Search filters can be provided. For example, a search filter in the form of a date range can be provided, for example, where the options are Most Recent and Specific Date Range. Using the Most Recent option, the user can enter a time and a scale of Minutes or Hours. Using the Specific DateRange option, the user will enter the start and end time. Using the Date Range option, Targets, Sources, and Filters can be specified. These allow the user to select what they want to see in this log browsing session. After the user selects the target, source, and applies any filters, they can start a browsing session to initiate the retrieval of logs from each target and ultimately display those logs on the interface.

[0160] The search query can be implemented in any suitable manner. In some embodiments, a natural language search process is performed to implement the search query. The search process can be used to perform searches across a dependency graph. Various relationships can be queried in the data, such as "runs on," "use by," "uses," and "member of," etc.

[0161] In some embodiments, the search query is a text expression (e.g., based on the Lucene query language). A user can enter a search query in the search box to search logs. The following are examples of content that can be included in a search query: (a) terms; (b) fields; (c) term modifiers; (d) wildcard search; (e) fuzzy search; (f) proximity search; (g) range search; (g) boost terms; (h) Boolean operators; (i) grouping; (j) field grouping; (k) escaping special characters.

[0162] A tabular view of search results can be provided. Some query refinement can be performed via table cells, allowing users to add / remove field-based conditions within the query text contained in the search box via UI actions. For example, when a user right-clicks a field, a pop-up window provides options for adding or removing conditions for filtering logs during the search. This is convenient for users to modify query text, and, using this approach, users can refine queries at the field level without having to know the internal field names.

[0163] There are many ways that can be provided to list fields in the search find results table for the user to select / deselect them for display purposes. One example approach is based on static metadata, and another possible approach is based on dynamic search results.

[0164] For list fields based on static metadata, use the basic field shuttle to list all defined fields. Some example fields that can be defined by log entry metadata include: (a) log file; (b) entry content; (c) rule name; (d) source name; (e) parser name; (f) source type; (g) target type; (h) target name. The values ​​of these fields can be obtained from the agent, while the log entry (although source, parser, rule, and target are all GUIDs / IDs) will need to be looked up when displaying.

[0165] For list fields based on dynamic search results, the first n (e.g., 10) fields will be shown, which will be suggested as the most important for the search. A "More Fields" link will trigger a pop-up window that lets the user select other fields. The user can see more information about those fields on the pop-up window than from the View menu. When listing fields, the system can use any suitable algorithm, for example, to assign a number to each field that is affected by how many rows in the search results have non-null values ​​or how many different values ​​exist across all search results for that field.

[0166] Given that there are so many dynamic fields available for users to select / deselect, it is desirable that users can save their field selections (field names and sizes). The system can store the last selected field, so when the user comes back to the page, he / she still gets the last selected field.

[0167] There may be a very large number (e.g., thousands) of log entries generated by a search, and users may not be able to browse through all of them to find the log of interest. For a specific search, users should be able to drill down into the details of the search findings with just a few clicks. In some embodiments, features include clickable bar charts and table paging. Using these navigation features, combined with customizable time ranges, users should be able to quickly jump to a point of interest. Accordingly, some embodiments provide drill-up from details to higher levels, so that users can easily navigate to the desired log entry via the bar chart. An example use case is that after a user drills down several levels, they may want to drill back up to the previous level to go down another bar. After a user identifies a log entry of interest through some searches, they may want to explore logs from a specific log source surrounding the log entry of interest, or explore logs from multiple log sources surrounding the log entry of interest in a time-interleaved manner. Some embodiments provide the option for users to page forward / backward through the logs surrounding a specified log entry. A graphical view of the search findings can be provided. This allows users to select fields to graphically render the results.

[0168] Some embodiments relate to improved techniques for addressing log distribution, trends, and correlations. For search results obtained from a particular search, the distribution can be based on log counts to give the user some high-level information about the logs. For each distribution type, the top n (e.g., 5 or 10) items are listed along with the number of logs found (where a "more..." link will result in a pop-up window listing all other items). When a user selects a particular item, only the logs corresponding to that item will be displayed in the table on the right, so the action is equivalent to filtering the search results by that item. This information can be presented in the following ways: (a) by target type; (b) by target, such as target owner and / or lifecycle state; (c) by log source; (d) by tag. In addition to showing the search results in a results table, the system can also provide the user with the option of switching between a table view and a corresponding distribution graph view.

[0169] In some embodiments, the results can be filtered by selecting a distribution item. The user can filter the results table by selecting one or more distribution items. By default, all distribution items are selected and all log entries are listed in the results table. After selecting one or more distribution items, the user can navigate the log entries via paging. With one or more distribution items selected, when the user clicks the search button to perform a new search, the selection of distribution items will be reset to select all distribution items.

[0170] Some embodiments provide a feature that shows trends in search results. Some embodiments provide a feature that shows correlations in search results. Regarding this feature, some embodiments provide launch links that allow users to navigate to search / view detailed logs when performing correlation analysis between events, metrics, and infrastructure changes. Launch links can be provided, for example, to allow users to navigate to an IT analytics product to analyze / view detailed events / metrics when they want to see a more general picture related to the logs.

[0171] Another feature of some embodiments relates to handling extended field definitions. Even when using the same baseline log type, individual log entries may contain inconsistent information from one log to the next. In some embodiments, this can be handled by defining base fields that are common to the log types, and then allowing for extended field definitions for additional data in the log entries.

[0172] To explain, consider a source definition that defines the log files to monitor. Log files are parsed into their basic fields based on the log type definition. Additional data that is inconsistent across all log entries can be extracted, such as Figure 13 As shown in 1300 of FIG. In this figure, the basic fields parsed from the log entry are Month, Day, Hour, Minute, Second, Host, Service, Port (optional), and Message. The goal is to extract the IP address and port from the second log entry. For example, since not every log entry has this structure, this goal may not be achieved in some implementations that are part of the log type. Here, the Message field for the second entry has the following content:

[0173] Accepted publickey for scmadm from xxx.xxx.1.1 port xyz ssh2 In some embodiments, the extended field definition on the Message field is defined using a format such as the following:

[0174] Accepted publickey for.*from{IP Address}port{Port}ssh2 For this log entry, two new fields IP Address and Port will be parsed out and will be available for reporting, searching, etc. This extraction occurs when the data is processed at the time of collection.

[0175] According to some embodiments, processing for implementing field definitions for processing time extensions includes: identifying one or more log files to be monitored, wherein some of the entries in the one or more log files may include additional data that is not present in or inconsistent with entries in other entries, such as an additional IP address field in one entry that does not appear in another entry; identifying source definitions for the one or more log files to be monitored; parsing the one or more log files into multiple basic fields using the source definitions; defining one or more extended fields for the one or more log files; and extracting the one or more extended fields from the one or more log files.

[0176] Therefore, some embodiments allow users to add extended field definitions. These are defined patterns that are seen within a field. A user can perform a create-like action on a source, and the source and all extensions will then become a new user-created source. Extended field definitions define new fields to be created based on the content in a given file field. In some embodiments, extended field definitions (and tags) can be applied retroactively. This allows processing past log data with later defined field definitions and tags.

[0177] Figure 14 Some example field definitions 1302 are shown. For the first case in the table, the user specifies to view the "Message" file field from the log entry and parsed by the file parser. This Message field will have text in it, but the user has identified that they want to capture the SIGNALNAME portion of the message as a new field for this specific message. This new field (SIGNALNAME) now becomes viewable in the captured log entry, viewable in the log browser, and can also be stored as part of the metric if a rule is created to do so. In this example, the extended field definition uses the entire content of Message. Users can bind either side of their expression with a wildcard pattern. For example, the definition can simply be "Send {SIGNALNAME}". The text shown is static text that is known to never change for this log message. Using [0-9]* in the expression means that any number of numeric characters can be located here, but they will only be ignored (because there is no associated field name to name this field). The text after the string "Send" will be assigned to the variable SIGNALNAME.

[0178] The last entry is another example where the user has defined two new fields, and in the first field, they have also defined how to use a regular expression to get this content. Here, before the period "." there is some character containing az, AZ, 0-9, or a hyphen. Everything that matches this expression should be added to a new extended field called HOSTNAME. Anything after the first period will be put into a new extended field called DOMAINNAME. The HOST field from the file parser still has all the content, but this extended field definition tells us that two new fields (HOSTNAME and DOMAINNAME) have been added in addition to the HOST field.

[0179] All extended field definitions that use {} delimiters to define new fields use parsing expressions. However, in this example, no parsing expressions are shown except for the HOSTNAME field in the last example. This is because in some embodiments, there is a default known regular expression pattern of (.)* that means any number of characters. If the user does not provide a regular expression, then this expression is used implicitly. If there is static text, then the system will get any characters between two pieces of static text. If there is no static text or characters after the field expression, then it is assumed that every character before the end of the file field is part of the value of the new extended field (such as DOMAINNAME in the last example and CONTENT_LENGTH_LIMIT in the third example). This may cause some problems if there are variations of this log entry that sometimes have additional text. The way to solve this problem is to also define a parsing regular expression for each field, rather than relying on the default implicit regular expression (.)*.

[0180] Some embodiments provide the ability to define regular expressions and save them with names. For example, the regular expression used above for host names is [a-zA-Z0-9\-]+.

[0181] An example of a saved regular expression could be:

[0182] IP_Address regular expression=>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

[0183] When referencing this saved regular expression in an extended field definition, the last entry in the table might instead look like this:

[0184] {HOSTNAME:@IP_Address}.{DOMAINNAME}

[0185] The new fields that will be created are HOSTNAME and DOMAINNAME. The referenced regular expression that is created and saved is called IP_Address. When the system performs processing on the proxy, it will replace the referenced regular expression "@IP_address" with the following regular expression string:

[0186] "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}"

[0187] By making minor changes to the input string from the user, the extended expression definition can be evaluated directly at the proxy (eg, using a Perl parsing engine).

[0188] In some embodiments, field reference definitions can be provided. This provides a feature where users can provide a lookup table for SQL queries to transform fields that may have unreadable values ​​into something more human-readable. Three example use cases highlight this need: (a) In a log entry, there may be an error code field (either a core field or an extension field) that has only numbers, in which case the user can provide a lookup reference that causes the system to add another new field to store a text description of what this error code means; (b) In a log entry, there may be a field (either a core field or an extension field) that has the GUID of the target, and the system can provide a lookup to the target table using an SQL query, which will create another new field that stores the display name of the target; (c) As a common use case, IP to hostname lookups can also be performed, where there may be IP addresses for clients in the log, where these IP addresses are used to look up the hostnames.

[0189] As noted above, log types may also be defined (also referred to herein as including "parsers" in some cases in this document) to parse log data. One example log type relates to a "log parser," which is a parser that can be used to parse core fields of a source. Another example log type relates to a "saved regular expression," which can be used when defining an extended field definition. For example, a hostname may be defined via a regular expression as "[a-zA-Z0-9\-]+." This regular expression can be saved with a name and then used at a later time when creating an extended field definition.

[0190] A log parser is a metadata definition of how to read a log file and extract the content into fields. Each log file can be described by a single parser to break each log entry into its basic fields. Log types can correspond to parsing expression fields, such as a Perl regular expression used to parse the file. When defining a log parser, the author identifies the fields that will always be present in the log file. In this case, the following are the fields that are present in each entry of the log file above:

[0191] Some fields can be very complex, meaning that the field will actually contain additional structured content that is used for some log entries but not for others. In some embodiments, this may not be handled by the log file parser because it will not be consistent across every line. Instead, when defining a source, you can define an extension field to break this field into more fields to handle these cases.

[0192] Profiles can be implemented for various constructs in the system, such as parsers, rules, and sources. Profiles capture differences between different uses and / or versions of data items and user products. For example, a source profile can be created that takes into account different versions of a monitored user product, for example, where the source profile changes the source definition between version 1 and version 2 of a monitored database. Rule profiles can be used to take into account differences in the rules to be applied. As another example, parser profiles can be provided to adjust parsing functionality, for example, due to differences in date formats between logs from different geographical locations. Different regular expressions can be provided for different parser profiles.

[0193] Regarding log entry delimiters, log files can have content that is known to always be one line per entry (syslog), or they can have content that can span multiple lines (Java Log4j format). The Log Entry Delimiter input lets you specify that the log file should always be parsed as one line per entry, or provide a header parsing expression that tells us how to find each new entry. The entry start expression will typically be the same as the first few segments of the parsing expression. The system uses this expression to detect when it sees a new entry rather than the continuation of the previous one.

[0194] For this example, the entry start expression could be:

[0195] ([A-Z]{1}[A-Z]{2})\s([0-9]{1,2})\s([0-9]{1,2}):([0-9{2}):([0-9]{2}) This expression looks for a strict month, day, hour, minute, second structure. If this exact sequence of characters is seen, then the "row" is considered the start of a new entry.

[0196] In some embodiments, a table corresponding to the parsed fields is maintained, and since the parsed expression is empty, the table starts out empty (no rows). As the user creates the parsed expression, defined fields are added to this table. This can be accomplished by monitoring the text entered in this field, and when a ")" is added, calling a function to determine how many fields have been defined. The system can ignore some cases of "(" and ")", for example, when they are escaped or when they are used with control characters.

[0197] For example, consider the following parsed language:

[0198] ([az]{2})\s([a-z0-9]+)

[0199] In this example, there are two pairs of "()" characters, which means that two fields are defined. What's inside is how the fields are found from the log entry - the UI that created the parser page for this doesn't care about what's inside the brackets. This is only evaluated and used on the proxy. The content outside the "(" and ")" is just static text that helps parse the line (this UI doesn't care about this either). To create the correct number of fields in the table, the method counts the number of "()" pairs in the parsing expression. For each field parsed out by the parsing expression, the user provides a field name based on one of the existing public fields.

[0200] Log parser construction

[0201] As noted above, log parsers are typically built in a manual process by someone who must be familiar with both the exact format of the log files to be analyzed and the specific programming infrastructure that will be used to implement the parser. There are a number of problems with this approach. For example, this highly manual process requires a significant amount of time and resources from skilled technicians, both upfront to create the parser and to maintain the parser on an ongoing basis in the face of potential changes in the log file format. Furthermore, this manual approach necessarily requires a priori knowledge of the log file format, which may not always be available before the log files begin streaming into the log analysis system. Finally, the lack of a suitable parser can potentially halt the log analysis pipeline from analyzing the affected log data.

[0202] Some embodiments of the present invention solve these problems by providing a method for automatically building a log parser. Instead of requiring a person to manually create the content of the log parser, the log content itself is used to build the parser.

[0203] Figure 15A high-level flow chart of a method for implementing this embodiment of the invention is shown. At 1502, one or more lines of a log file are received for processing. In some embodiments, a log parser is built as each line of the log file is received in a streaming manner. This approach allows the parser to be built in real time as the contents of the log file are received. In an alternative embodiment, several lines from the log file can be collected together before processing the lines. This approach can be useful for performing batch processing on log file lines, for example to implement certain types of processing, such as clustering or grouping analysis of a collection that may require a minimum number of log file lines before processing.

[0204] At 1504, lines from the log file are analyzed. Analysis is performed to identify specific content and differentiating segments within the log file lines. As additional lines are processed and more information about the lines is obtained, a higher level of certainty about the basic structure of the log file lines can be obtained. It is to be noted that the number of lines that need to be reviewed in order to generate an accurate parser depends on the complexity and content of the log file lines. However, by using the techniques described herein, many log files may only require 10-20 lines (or even fewer lines) to be analyzed to build an acceptably accurate log parser. This ability to generate a log parser based on viewing a relatively small number of lines allows the log parser generation process to be performed in a very time-efficient manner, and therefore improves the operation of the computing system itself because it allows the log parser generation process to be performed in real time as the log file data is streamed into the log analysis system.

[0205] At 1506, a parser is then built based on the analysis of the lines from the log file. This is performed, for example, by scanning the contents of one or more sets of logs to build regular expressions for parsing the logs. This embodiment operates by traversing the selected set of lines to identify commonalities between the lines and then building regular expressions that can be used to generally parse log files containing similar lines of log entries.

[0206] Figure 16 A more detailed flow chart of a method for implementing this process according to some embodiments is shown. At 1602, a master list is created from the first line of the log file being analyzed. The master list includes a mapping structure that maps the contents of the log file line to the identified element types within the line. Examples of these element types include, for example, numeric / integer types, character types, and string types.

[0207] Once the first line has been processed, another line from the log is selected for analysis at 1604. At 1606, the analysis is performed by moving through the line being analyzed to compare against the master list. This action is performed to identify the variable and immutable parts of the (one or more) lines being analyzed. This can be performed by starting at the beginning of the line and moving forward until there is a mismatch. At this point, the process finds the next (or next multiple) common characters. One of the identified common characters is considered a "delimiter", so the middle range is marked as variable. It should be noted that the size of the middle range can be variable between two lines, so the algorithm should be robust enough to handle this problem. An example algorithm for identifying the common part that should be considered a delimiter is described in more detail below. The process loops until the end of the line is reached.

[0208] The master row can then be updated to reflect the common portion and the variable portion at 1608. Additionally, the value of the variable portion can be stored if desired.

[0209] At 1610, it is determined whether there are any additional lines to analyze. In some embodiments, it is not necessary to analyze every line in the log to perform analysis of the log. Instead, only a subset of lines (e.g., 10 lines) need to be analyzed to build the parser. If additional lines need to be analyzed (e.g., only 2 of the 10 lines for analysis have been processed), then the process returns to 1604 to 1608 to select and analyze another line from the log.

[0210] If no additional rows need to be analyzed, then at 1612, the updated primary row is ready to be processed. As described in more detail below, one example type of processing is to assign, for at least one of the variable portions, the at least one variable portion to the least restrictive data type that encompasses the variability of the values ​​detected in the at least one variable portion. Additionally, commonalities can be identified between rows so that regular expressions can then be constructed from the commonalities. Regular expressions can be generated for immutable portions with placeholders to implement a log parser, where at least two different placeholders are associated with different data types.

[0211] Figures 17-1 to 17-21 A graphic illustration of this process is provided. Figure 17-1 An example of a log file 1702 is shown. Two lines are shown in the figure within the log file 1702. The first line includes the content "N=Bob." and the second line includes the content "N=Sue."

[0212] The first action is to select line 1 from the log file 1702 to build the master list 1704. Figure 17-2As shown in FIG. 17, row 1 corresponding to "N=Bob." is selected for processing.

[0213] To construct the master list 1704, each portion / cell of the contents within row 1 is examined to identify the cell type associated with that portion of the row (also referred to herein as "parsing a cell"). According to one embodiment, each portion of the row is identified according to one of the following parsing cell types: (a) string - this is the default parsing cell type corresponding to any type of element that can exist within a string; (b) alpha - this parsing cell type corresponds to any number of consecutive alpha elements; (c) integer - this parsing cell type corresponds to any number of consecutive integer elements; and / or (d) field rule type - this parsing cell type corresponds to a type identified based on a rule definition, and can be associated to a complex combination of any number of characters, integers, or symbols. The greater the restriction of the type, the more advantageous it is to select that type for the element(s) within the row.

[0214] Figures 17-3 to 17-8 The process of constructing the master list 1704 for row 1 of the log file 1702 is shown. As Figure 17-3 As shown in FIG. 17, the first character "N" is retrieved and placed in the first position in the master list. A parsing cell type is also identified for this character. In this case, the initial parsing cell type of "string" is assigned to this character, as the process does not yet have enough information to know whether the variability of this element within multiple rows should cause this element to be assigned to a different parsing cell type. Thus, since the master list is concerned with the first row of the log file, this element (as well as each of the other elements within the row) will be assigned to the default "string" parsing cell type, as this parsing cell type encompasses every possible type of element that can exist in a row. As Figure 17-4 As shown in FIG. 17, the next character from row 1, "=", is also placed in the master list 1704, and is assigned to the parsing cell type "string". As Figure 17-5 As shown in FIG. 17, the next character from row 1, "B", is also placed in the master list 1704, and is assigned to the parsing cell type "string". Figure 17-6 As shown in FIG. 17, the next character from row 1, "o", is placed in the master list 1704 and is assigned to the parsing cell type "string". Figure 17-7 Similarly, as shown in FIG. 17, the next character from row 1, "b", is placed in the master list 1704 and is assigned to the parsing cell type "string". Finally, as Figure 17-8 As shown in FIG. 17, the last character from row 1, ".", is placed in the master list 1704 and is assigned to the parsing cell type "string".

[0215] Next, as Figure 17-9As shown in , row 2 corresponding to "N=Sue." is retrieved from log file 1702 and compared against master list 1704. The contents of row 2 can be analyzed on an element-by-element basis relative to master list 1704. To illustrate this type of analysis, Figure 17-10 The contents of row 2 are shown organized on an element-by-element basis.

[0216] Figures 17-11 to 17-15 Such a comparative analysis is shown between row 2 and the main list 1704 . Figure 17-11 1704. Here, the master list 1704 includes an "N" in the first position, which matches the element "N" in the same position within row 2. Thus, this shows that the master list 1704 correctly indicates that the first element of the row has "N" as its content. Similarly, Figure 17-12 Analysis of the second element position is shown, where the master list 1704 includes a "=" in the second position, which matches the element "=" in the same position within row 2. This indicates that the master list 1704 correctly shows that the second element of the row has "=" as its content.

[0217] However, if Figure 17-13 , a comparison of the third element position indicates a difference between the contents of the main list 1704 and the contents of row 2. In particular, the main list 1704 has a "B" in the third element position, while row 2 includes an "S" in the third element position. This indicates that the third element position is a variable portion of the row(s).

[0218] The process then continues to identify the next (or next multiple) common elements that should be considered as delimiters between the common part and the variable part. In the current example, the "." element in the sixth element position is the next common element. Figure 20 A method that can be used to identify the next common element that should be considered a delimiter is described in more detail. It is noted that this method of "skipping ahead" to find the next common portion allows any varying number of characters within each of a plurality of lines to be compared, since it does not matter how many characters of each line are skipped to identify the next common character.

[0219] like Figure 17-14 , the common portion is the first element position ("N"), the second element position ("="), and the sixth element position ("."). The variable portion includes the third element position ("B" in the main list and "S" in row 2), the fourth element position ("o" in the main list and "u" in row 2), and the fifth element position ("b" in the main list and "e" in row 2).

[0220] The variable portion forms the scope of the analysis, in which its contents can be analyzed as a collective group of elements. Further, within the variable portion, common parse unit types can collapse together, e.g., for the variable portion of both the main list and row 2, this corresponds to "Bob" for the main list and "Sue" from row 2. The most restrictive parse unit type associated with these values is the alpha parse unit. Thus, as shown in Figure 17-15 each of the string values for the variable portion of the main list are replaced with an alpha parse unit.

[0221] When considering the contents from both the main list and row 2, the parse unit definitions within the main list 1704 can also be used to track the specific contents from each of the rows that have been analyzed. Here, the "Bob" and "Sue" values from row 1 and row 2 for this element position can be included within the parse unit definitions for the alpha parse units within the main list. This results in the main list 1704 as shown in Figure 17-16 When considering the contents from both the main list and row 2, the parse unit definitions within the main list 1704 can also be used to track the specific contents from each of the rows that have been analyzed. Here, the "Bob" and "Sue" values from row 1 and row 2 for this element position can be included within the parse unit definitions for the alpha parse units within the main list. This results in the main list 1704 as shown in

[0222] Within a row of a log file, there can be sections of content that can be interpreted as having a meaningful label / type associated with it. For example, a series of numbers separated by "." values (such as "123.45.67.89") can be recognized as an IP address. Thus, instead of generally representing such sequences as integer, alpha, or string parse units, a "field rule" type can be constructed that associates meaningful labels to these types of sequences. The field rule type can include a rule definition associated with the combination of characters, integers, and / or symbols that are associated with a given sequence of interest.

[0223] Figures 17-17 to 17-19 This process of identifying a field rule parse unit for the main list is shown. As shown in Figure 17-17 there can be any number of field rules 1710 that have been defined for the log analysis system. Each of these field rules corresponds to a different type of sequence for which it is of interest to identify a meaningful type for the sequence. Examples of field rule types include field rules for IP addresses, timestamps, identifiers, etc.

[0224] Figure 17-18An example field rule 1712 is shown that may be applied to a variable portion of a master list. Field 1712 corresponds to a "Name" type (or name / label), for example, where the field rule may be applicable to identifying a person's name. Field rule 1712 may be associated with a regular expression 1714 to identify the portion of the master list to which the field rule is relevant. Here, regular expression 1714 is "[AZ][az]+". This regular expression corresponds to any sequence of characters where the first character is an uppercase letter (from AZ) followed by any number of subsequent non-uppercase letters (from az). Here, the recorded data set for the alpha parsing unit for the master list (e.g., "Bob" and "Sue") matches the regular expression of this "Name" field rule. As Figure 17-19 As shown in , the master list 1704 may thereafter be updated to replace the alpha parsing unit type(s) with the field rule parsing units.

[0225] It is noted that processing of the master list to identify field rules can be performed in a post-processing action after the master list is constructed (after parsing multiple lines from the log file). Alternatively, field rules can be identified as lines are analyzed separately in a streaming manner. In another embodiment, field rules can be identified both during processing of the lines and afterwards in a post-processing step.

[0226] In some embodiments, field rule processing is performed only for sections of a line that are identified as variable. Alternatively, field rules may relate to both constant and variable portions of a line.

[0227] Once a sufficient number of lines from the log file have been processed, regular expressions can be built from the master list. For example, Figure 17-20 As shown in , a regular expression "N=([AZ][az]+)\." can be constructed that is correlated with the values ​​maintained in the master list 1704. In particular, the constant portion of the master list can be directly inserted into the regular expression. The portion of the regular expression that corresponds to the field rule can be pulled from the regular expression definition of the field rule itself.

[0228] Finally, if Figure 17-21 As shown in , a log parser 1722 can be constructed for a log file. Log parser 1722 can include a regular expression constructed from a master list, where the regular expression is associated with a line definition for the log file, where the line definition for the log file can be used to read and parse each line from the file. Other items of data / metadata can be included with log parser 1722. For example, log parser 1722 can be associated with a parser identification number / name and / or with a file type identifier.

[0229] One key advantage of this approach is that it can efficiently detect sub-patterns that vary from one another, the approach separately matches high-level patterns and then uses sub-pattern detection to attempt to characterize variable portions that are not fixed parts of the high-level patterns. One example sub-pattern is a key-value pair.

[0230] This approach can also be used to define skeleton portions to build regular expressions and to build parsers that can assign portions of the expression to variables. This approach can also handle patterns that fall below a similarity threshold by assigning variable portions to keep items in the same log consistent, if possible. In some embodiments, parsers are generated for future processing, not just for classification.

[0231] By using this approach, logs of any level of complexity can be processed to build log parsers. Consider the following log entries, which are a slightly more complex example:

[0232] [11.22.33.44] Name = Bob Age = 27

[0233] [10.20.30.40] Name = Sue Age = 30

[0234] This embodiment operates by traversing the selected set of rows to identify commonalities between the rows and then building regular expressions that can be used to generally parse log files containing similar rows of log entries. The master list can be built from the first row, with the first row compared to the second row. The analysis identifies variable and non-variable portions of the row(s) being analyzed.

[0235] Assume that the first row (with name = Bob) is used to initially build the master list. The next row (with name = Sue) is then analyzed to identify variable and non-variable portions between the two rows. Here, the first portion of the rows has the same common character "[". Moving from this character, it can be seen that there is a middle range of different values until the right square bracket "]" is reached. The middle range can optionally be considered to include a common value for the "." character. The rest of the rows can similarly be analyzed so that the "Name = " portion and the "Age = " portion are found to be common portions, while the range of characters after these common portions are found to be variable portions. The master row can be updated to reflect the common portions and the variable portions. In addition, the variable values can also be stored, if desired.

[0236] The updated master row can then be processed to build regular expressions from the commonalities. For the example rows above, the following identifies the common portions, the variable portions, and the values of the variable portions.

[0237] [{Variable Segment 1}] Name = {Variable Segment 2} Age = {Variable Segment 3}

[0238] Variable segment 1: {11.22.33.44, 10.20.30.40}

[0239] Variable segment 2: {Bob, Sue}

[0240] Variable segment 3: {27, 30}

[0241] This can then be used to construct an appropriate regular expression to parse the lines from the log. For example, the first part of a line of an IP address may correspond to the following regular expression: "\[[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\.\]". Note that a field rule (defined to include this regular expression) can also be used to associate this part of the line with the IP address.

[0242] According to some embodiments, row preprocessing may be performed in advance to prepare the log data for processing. To explain, consider Figure 19 The example log file 1902 shown in FIG. This log file 1902 includes four lines, where the first line includes "N=", the second line includes "Bob.", the third line includes "N=", and the fourth line includes "Sue.". Figure 17-1 The exact same content exists on two lines in file 1702, but is spread across four lines in file 1902. However, the problem is that the log parser generator, which expects each line to be processed separately as a single unit, fails when faced with Figure 19 The log file structure shown in log file 1902 will fail because each log entry actually covers two lines at a time (for example, the first two lines as the first log entry, and the third and fourth lines as the second log entry).

[0243] Figure 18 A process flow of an embodiment that addresses this and other types of non-standard row format situations is shown.At 1802, one or more rows are identified from a log file for pre-processing.

[0244] At 1804, the rows are analyzed for grouping purposes. One possible approach that can be taken to group the rows together is to examine the rows' timestamps. For example, in some systems, it is possible that multiple rows that are related to each other only contain a timestamp for the first row. In this case, the rows are grouped together until another timestamp is identified. Even if each row includes its own timestamp, the commonality of the timestamp values ​​allows the multiple rows to be identified as part of a single whole. As another alternative, clustering can be performed to cluster together groups of rows that are assumed to be linked to each other. Another possibility is to perform pre-sorting on the rows to identify the row structure in order to identify rows that should be grouped together.

[0245] Once the groups have been identified, the grouped rows can be considered together for log analysis purposes. According to 1806a, one possible approach is to manipulate the rows so that the grouped content appears within a single row. At 1806b, another approach is to group multiple related rows into a single log entry for analysis purposes.

[0246] Figure 19 Section 1904a shows a method of manipulating grouped content so that the content appears within a single line. Here, the original file 1902 includes four lines, where the first line includes "N=", the second line includes "Bob.", the third line includes "N=", and the fourth line includes "Sue.". The intention is to group the first and second lines together with each other, while the third and fourth lines form another grouping. As shown in section 1904a, the contents of the first and second lines are combined to form a single line with the content "N=Bob.". Similarly, the contents of the third and fourth lines are combined to form another single line "N=Sue.". Each of these newly formed single lines is then processed for generating a log parser.

[0247] In an alternative approach, shown in section 1904b, no new lines are created to group together multiple related lines. Instead, for analysis purposes, the multiple lines are simply logically grouped together as a single log entry. As shown in section 1904b, a first logical log entry 1906s is formed from the first two lines, and a second logical log entry 1906b is formed from the third and fourth lines. The master list described above is constructed by traversing the elements of the two lines that relate to a common logical log entry. In this case, the "newline" character that separates lines within a single log entry can be considered simply another character to be recognized and processed within the master list for a given entry.

[0248] As noted earlier, when building a log parser, contiguous segments of a log line can be treated as units. Delimiters within a line can be identified to determine contiguous segments to be identified. Each line can be considered for its common portion and variable portion, where the common portion can correspond to a delimiter that separates one or more elements within the variable portion as a sequence of elements for analysis. For example, in the first line of the log file 1702 shown in Figure 17-1 the variable portion "Bob" is bounded on the left by the common portion "N=" and on the right by ".". Similarly for line 2, the variable portion "Sue" is bounded on the left by the common portion "N=" and on the right by ".". Thus, the common element "." can be identified as a delimiter that separates the variable portion (e.g., "Bob" or "Sue") that precedes it from other portions of the line for analysis.

[0249] However, if one or more of the common portions should actually be considered part of a content segment that should be analyzed as a unit, the simple solution of identifying only the common portions as delimiters fails. To illustrate this problem, consider the log file 2102 shown in Figure 21-1 Here, line 1 and line 2 have a common portion at the fourth element position (the letter "o" in both lines at that position) and a common portion at the sixth element position ("." in both lines at that position). As a human observer, it is fairly easy to determine that "." should be the delimiter (rather than the "o" character), since the letter "o" forms part of the name for both the "Bob" and "Tod" sequences of letters. However, when performing automated processing of the lines, especially when faced with these lines without prior knowledge of the line content, it is much more difficult problem to identify the correct delimiter. This is especially true because the automated processing system can not have prior knowledge of the "name" type as part of the line structure. In fact, even without prior knowledge of the line structure, the processes described herein will be used to discover such types within the log line structure.

[0250] According to some embodiments, an inventive method is provided to identify which of one or more common elements within a line should be treated as a delimiter. The method operates by traversing the line to identify common elements, where a score for each common element is determined considering a combination of element position and element weight. The element with the highest score (or lowest score, depending on how the score is computed) can then be identified as the delimiter.

[0251] Figure 20A flowchart showing a method for efficiently identifying positive delimiting element within a set of log content, according to some embodiments of the present application, is shown. At 2002, the row content(s) for at least two rows are traversed (e.g., from left to right within the row) to identify common elements that delimit a variable portion of the row.

[0252] At 2004, the process then traverses the rest of the row to identify additional common element(s) within the row. There can be any number of additional common element(s).

[0253] At 2008, a score is computed for each of the identified common elements. First, the position of the common element within the row is determined for the score. The general idea is that a possible delimiting found earlier (e.g., closer to the left when traversing from left to right within the row) should be the first delimiting to consider, all other conditions being equal. For example, consider the following row: "Names: Bob Joe Sam". In this example row, there is a first space between "Bob" and "Joe", and a second space between "Joe" and "Sam". In this case, both spaces can be delimiters, but the first space (the left space between "Bob" and "Joe") should be identified first. Only after that, when the delimiting identification process is run again, the second space (between "Joe" and "Sam") will be identified as the next delimiting. Thus, when choosing between the first and second spaces, the position of the first space should receive a more prominent score than the second space. This is done at 2010 by providing a score factor determined by the position of the element within the row. For example, the sum or average of the positions of the elements can be identified and associated with the element.

[0254] Furthermore, the type of the found element should also be taken into account in the delimiting score for the element. This is done at 2012 by applying a weighting factor to the identified common element(s) within the row. Figure 22 A chart 2202 is shown with some example weights that can be applied to the identified common elements in some applications of the present application. In particular, this chart is based on the assumption that the lower the score, the more likely it is that the element is a delimiting. Thus, Figure 22Each type of element shown in is associated with a weighting factor, where element types that are more likely to be delimiters have smaller weighting factors, and element types that are less likely to be delimiters are associated with larger weighting factors. In this example, a space is likely to be considered a delimiter; therefore, as shown in row 2206, this element type is associated with a very small weighting factor. On the other hand, alphanumeric characters are considered to be one of the least likely element types to be delimiters; therefore, as shown in row 2210, this element type is associated with a very high weighting factor.

[0255] The weighting factor can also be associated with a more complex rule that takes into account the combination of elements. For example, since a typical IP address has a sequence of digits dotted with "." elements, this means that the "." set between two integer sequences is unlikely to be a delimiter, but more likely to be part of the IP address field. In addition, non-integer numbers (e.g., floating point numbers) can include a decimal point between two digits (e.g., for the "." element between the two digit elements in "2.3"), which also makes it unlikely that the "." element is a delimiter in this case, but more likely to be part of the numerical value. Therefore, as shown in row 2208, this combination of elements can be associated with a rule that identifies the "." element between two integers, wherein this rule is associated with a weighting factor so as to deviate from the "." as a delimiter in this type of element combination.

[0256] As another example rule, consider the case where a given character has already been found to be a delimiter in a row. In this case, the same character that follows it in the same row is likely also a delimiter. For example, a "=" element that was previously found to be a delimiter for a key-value pair in a row is likely also a delimiter for another key-value pair later in the same row. Therefore, as shown in row 2204, a rule for this situation can be associated with a weighting factor to heavily favor elements that were previously identified as delimiters being considered delimiters again.

[0257] Once the scores are calculated, delimiters can be identified by comparing the scores of different possible delimiters at 2014. For example, if the scoring is configured so that lower scores correspond to greater likelihood of being a delimiter, then the element in the row(s) with the lowest calculated score will be identified as a delimiter. This process can then be repeated to identify any number of additional delimiters within the row, if any exist.

[0258] It should be noted that while this example calculates the delimiter score where the lowest score is most likely to be a delimiter, the application of the principles disclosed herein can also be operated to calculate the score where the highest score corresponds to the most likely delimiter. In this alternative approach, the weighting factors will be configured so that the element type that is most likely to be a delimiter will be associated with a higher weighting factor, and the element type that is less likely to be considered a delimiter will be associated with a lower weighting factor.

[0259] Figures 21-1 to 21-5 The delimiter recognition process is shown. Figure 21-1 Log file 2102 is shown, where line 1 corresponds to "N=Bob." and line 2 corresponds to "N=Tod." Figure 21-2 As shown in , the common element "=" on the left is found in both rows. The process then traverses the remaining rows to identify more common elements. Figure 21-3 As shown in , it can be seen that "o" is the first common element and "." is the second common element.

[0260] A delimiter score will then be calculated for each of the two common elements. Figure 21-4 The process of calculating the score for each of these elements is shown. First, the relative position of the element within the row is calculated for each row. For the element "o", this element is present at position 1 for every row. Therefore, the sum of these position values ​​is 2. For the element ".", this element is present at position 3 for every row. Therefore, the sum of the position values ​​is 6.

[0261] Next, a weighting factor is identified for each element. Figure 22 Example weighting factors that may be used in the current score calculation are shown. For element "o", this is an alphanumeric character that is Figure 22 The weighting factor 100 is associated with the element ".". Figure 22 The weighting factor in the diagram is associated with 1.

[0262] In this example, the score is calculated by multiplying the weighting factor by the sum of the positional locations for the element. For the "o" element, the score would therefore be positional sum (2) * weighting factor (100) = delimiter score 200. For the "." element, the score would be positional sum (6) * weighting factor (1) = delimiter score 6.

[0263] The scores are then compared to identify the lowest score, where the element with the lowest score is considered a delimiter. Here, the "." element has a lower score than the "o" element (6 < 200). Therefore, the "." element is identified as a delimiter.

[0264] like Figure 21-5, it is now possible to identify the parsed segments "Bob" and "Tod" between the "=" element and the delimiter "." In this case, identifying "." (rather than "o") as the delimiter produces the correct result because it can be seen that although the "o" character is a common element between the two lines, the "o" character is actually part of the parsed segment and not a delimiter.

[0265] Another technique that can be applied in some embodiments is to automatically perform key-value extraction from log data. This approach is particularly useful, for example, for implementing the extended field definitions described above. The current embodiment implements this by identifying the first and last key-value pair delimiters in a line and then using a split function to process the content in between to extract the key-value data.

[0266] Figure 23 A flow chart of an example method for performing key-value extraction is shown. The process begins at 2302 by parsing a line and identifying the first key-value delimiter it sees. The key-value delimiter can be, for example, a "=" character. If no appropriate key-value delimiter is found in the line, the process exits.

[0267] Next, at 2304, an attempt is made to find a key-value (KV) pair separator (if a KV separator is found, then break). Such a pair separator can be, for example, a space between different key-value pairs. The process loops through this step to find additional pair separators. This thus identifies the range of key-value pairs present within the row for processing.

[0268] The process then returns to the beginning of the identified range to extract the key-value content. At 2306, the key to the left of the first key-value delimiter is identified. At 2308, the value to the right of the last KV delimiter is identified. At 2310, the identified portion of the line is then parsed to identify the key value. For example, the "split" function from Java or Perl can be used to perform this action. For the current line, this action therefore identifies the key value for each key-value pair in the line. Therefore, this method can be used to automatically perform key value extraction. The process traverses the remaining key-value pairs in the identified range to extract the key-value data for all key-value pairs.

[0269] Figures 24-1 to 24-12 Iterate this process. Figure 24-1 An example line 2402 from a log file is shown with the following content: "11 / 12 / 2017 Name=Bob ID=5 Age=21". Figure 24-2 As shown in , the process begins by parsing line 2402 and identifying the first key-value separator seen. The key-value separator can be, for example, a "=" character. Here, searching for the first key-value (KV) separator will find the "=" character within "Name=Bob".

[0270] Next, if Figure 24-3 As shown in , an attempt is made to find a key-value pair separator. This separator can be, for example, a space between different key-value pairs. In line 2402, this identifies a space between the key-value pair (KVP) "Name=Bob" and the next KVP "ID=5". The process loops through this step to find additional pair separators. For example, Figure 24-4 As shown in , another pair separator is found between KVP "ID=5" and the next KVP "Age=21". Figure 24-5 As shown in FIG, there is no other key-value pair that is an identifier to the right of the KVP "Age=21" in row 2402.

[0271] At this point, the scope of the analysis has been identified for the key-value content extraction process. Figure 24-6 As shown in , the analysis range is from KVP "Name=Bob" to KVP "Age=21".

[0272] Now processing will occur to extract the "key" and "value" for each key-value pair. Figure 24-7 As shown in , identify the key to the left of the first KV separator. In the example row, the first KV separator is between "Name" and "Bob". The key to the left of this separator is "Name". Next, as Figure 24-8 As shown in , the value to the right of the KV delimiter is identified. In the example row, the value to the right of the delimiter is "Bob". These values ​​can be recorded in any suitable structure such as a database table or a key-value data structure.

[0273] Then processing proceeds to the next key-value pair. Figure 24-9 As shown in , identify the key to the left of the next KV separator. In the example row, the KV separator is between "ID" and "5". Therefore, the key to the left of this separator is "ID". Next, as Figure 24-10 As shown in , the value to the right of the KV delimiter is identified. In the example row, the value to the right of the delimiter is "5". These values ​​are recorded in the appropriate storage structure.

[0274] Then processing proceeds to the last key-value pair. Figure 24-11 As shown in , identify the key to the left of the last KV delimiter. Here, the last KV delimiter "=" is between "Age" and "21". The key to the left of this delimiter is "Age". Next, as Figure 24-12 As shown in , the value to the right of the KV delimiter is identified. In this case, the value to the right of the delimiter is "21". These values ​​are recorded in the appropriate storage structure. At this point, all key-value pairs have been identified and extracted into the key-value storage structure.

[0275] Additional optimizations can be performed to efficiently extract key-value content from log files. To explain, again consider the log file line 2402 shown in Figure 24-6 The analysis section that has been identified for this line 2502 is fairly easy to process, as it contains only key-value pairs to be processed. Thus, iterative application of the method described above to check the key and value on either side of the KV delimiter " = " will result in the correct key / value data for each key-value pair.

[0276] However, as shown in line 2504 in Figure 25-1 there is the possibility that non-key-value content can exist within the boundaries of the analysis section. In this case, the date content "11 / 12 / 2017" appears within the analysis section. Thus, a simple application of the process of checking the value on the left / right side of the KV delimiter " = " can end with an incorrect assignment of this content to the value content of the previous KV pair (e.g., where the key is "Name" and the value of this key is erroneously identified as "Bob 11 / 12 / 2017") or to the subsequent key content (e.g., the key is erroneously identified as "11 / 12 / 2017 ID" and the value of this key is identified as "5").

[0277] This problem cannot be corrected simply by considering spaces characters as delimiters for identifying keys and values. This is because certain keys and / or values can be intended to include spaces as part of the key / value content. To explain, consider the line 2506 shown in Figure 26-1 In this line, the first KV pair has "Name" as the key. The value of this key is assumed to be "Bob Smith" with spaces between "Bob" and "Smith". Thus, any method that always considers spaces as delimiters will incorrectly assign only the "Bob" portion as the value, and not incorrectly identify "Smith" as part of the value content.

[0278] One possible method to address this problem is to perform pre-processing to classify portions of the line so that the key-value portions of the line can be identified for key-value extraction. With this method, Figure 25-1 the non-KV portion of line 2504 in can be correctly identified (e.g., as a date field), and the key-value extraction process described above will thus ignore this portion of the line. As shown in Figure 25-2 this will result in multiple analysis sections under KV extraction. This method can similarly be used to address Figure 26-1As shown in line 2506, the first KV pair in line 2506 has "Name" as the key, and the value of this key is assumed to be "Bob Smith", and there is a space between "Bob" and "Smith". Any method that treats the space as a delimiter will incorrectly assign only the "Bob" portion as the value, and will not correctly identify "Smith" as part of the value content. However, preprocessing can be used to recognize the fact that spaces can exist within the value field for this key-value pair, and as Figure 26-2 As shown in , the correct portion of the row will be identified for each key-value pair, where the space element between "Bob" and "Smith" is considered to be part of the value field rather than a delimiter. The method described in U.S. application serial No. 14 / 863136 filed on September 24, 2015 can be used to identify pattern signatures to implement this method, and the entire contents of the U.S. application are incorporated herein by reference. The pattern signature can correspond to the KV signature (e.g., an identified KV delimiter (such as "=") separates the identifiable key and value portions), which is used to identify the KV analysis portion of the row (and ignore the non-KV portion of the row).

[0279] Another approach is to perform post-processing to correct any problematic assignments of content to key or value fields. This approach can be used, for example, to check for incorrect type(s) of the value(s) within a key-value field. To explain, consider Figure 25-1 Consider row 2506 shown in , and consider that a simple application of the process of only checking the values ​​to the left / right of the key-value delimiter "=" may end up with the incorrect assignment of date content to the value content of the preceding key-value pair, for example, where the key is "Name" and the value of this key is incorrectly identified as "Bob 11 / 12 / 2017". The log analysis system can be configured with a rule that limits the scope of type elements to fields that can be identified as "Name" fields, which excludes the " / " character from valid names. In this case, post-processing will be able to identify the incorrect portion of the value field based on the " / " character within the "Bob 11 / 12 / 2017" field. The analysis system can then choose to exclude the entire extracted key / value content, or choose to correct the incorrect content. Correction can be achieved by scanning forward and / or backward within the row content to identify the correct set of content to be assigned to the key and / or value fields. In this case, the "Bob 11 / 12 / 2017" field will be corrected to associate "Bob" only with the value field of the key "Name".

[0280] Thus, what has been described are improved systems, methods, and computer program products for implementing log analysis methods and systems that can efficiently configure, collect, and analyze log records. In particular, an improved method for automatically generating a log parser by analyzing the line content of a log has been described. Furthermore, an efficient method for extracting key-value content from log content has been described.

[0281] System Architecture Overview

[0282] Figure 27 is a block diagram of an illustrative computing system 1400 suitable for implementing embodiments of the present invention. The computer system 1400 includes a bus 1406 or other communication mechanism for communicating information that interconnects subsystems and devices such as a processor 1407, system memory 1408 (e.g., RAM), static storage device 1409 (e.g., ROM), disk drive 1410 (e.g., magnetic or optical), communication interface 1414 (e.g., a modem or Ethernet card), display 1411 (e.g., CRT or LCD), input device 1412 (e.g., keyboard), and cursor controller.

[0283] According to one embodiment of the present invention, computer system 1400 performs specific operations by executing one or more sequences of one or more instructions contained in system memory 1408 by processor 1407. Such instructions can be read into system memory 1408 from another computer-readable / usable medium (such as static storage device 1409 or disk drive 1410). In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions to implement the present invention. Therefore, embodiments of the present invention are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" refers to any combination of software or hardware used to implement all or part of the present invention.

[0284] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 1407 for execution. Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as disk drive 1410. Volatile media include dynamic memory, such as system memory 1408.

[0285] Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, cloud-based storage, or any other medium from which a computer can read.

[0286] In an embodiment of the present invention, the execution of the instruction sequences for practicing the present invention is performed by a single computer system 1400. According to other embodiments of the present invention, two or more computer systems 1400 coupled via a communication link 1415 (e.g., a LAN, PTSN, or wireless network) can coordinate with each other to execute the instruction sequences required to practice the present invention.

[0287] Computer system 1400 can send and receive messages, data, and instructions, including programs, i.e., application code, via communication link 1415 and communication interface 1414. Received program code can be executed by processor 1407 as it is received and / or stored in disk drive 1410 or other non-volatile storage for later execution. Data can be accessed from database 1432 maintained in storage device 1431, which is accessed using data interface 1433.

[0288] In the foregoing description, the present invention has been described with reference to specific embodiments of the present invention. However, it will be apparent that various modifications and changes may be made to the present invention without departing from the broader spirit and scope of the present invention. For example, the process flow described above is described with reference to a specific order of processing actions. However, the order of many of the processing actions described may be changed without affecting the scope or operation of the present invention. Accordingly, the description and drawings should be considered illustrative rather than restrictive. In addition, the illustrated embodiments do not necessarily have all the aspects or advantages shown. The aspects or advantages described in conjunction with a particular embodiment are not necessarily limited to that embodiment and may be practiced in any other embodiment, even if not shown as such. Moreover, throughout this specification, references to "some embodiments" or "other embodiments" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments are included in at least one embodiment. Therefore, the appearance of the phrases "in some embodiments" or "in other embodiments" throughout various places in this specification does not necessarily refer to the same one or more embodiments.

Claims

1. A method for implementing a log parser in a log analysis system, comprising: In response to receiving log data comprising a plurality of log entries: storing a first copy of the log data in an indexed data repository comprising a plurality of partitions; indexing the log data in the indexed data repository based on a plurality of time ranges respectively associated with the plurality of partitions; The plurality of log entries are grouped based on parameters associated with the log data to generate a second copy of the log data, wherein the parameters include one or more of the following: a first set of identifiers identifying respective tenants associated with the plurality of log entries; a second set of identifiers identifying respective targets within the computing environment from which the plurality of log entries were obtained; and a third set of identifiers identifying respective sources where respective destinations store the plurality of log entries; storing a second copy of the log data grouped based on the parameter in a historical data repository; wherein the historical data repository further includes historical log data associated with times older than the plurality of time ranges in the indexed data repository; wherein the indexed data repository and the historical data repository are separate; Perform at least the first log query by doing the following: determining a first time range associated with the first log query; In response to determining that the first time range is within one or more time ranges of the plurality of time ranges: selecting the indexed data repository instead of the historical data repository to perform a first log query; applying a first log query to one or more partitions of the plurality of partitions corresponding to the one or more time ranges; Perform at least the following second log query: determining a second time range associated with the second log query; In response to determining that at least a first portion of the second time range is not within the plurality of time ranges: selecting the historical data repository instead of the indexed data repository to perform a second log query; applying a second log query to the historical data repository; The method is performed by at least one device including a hardware processor. 2 . The method of claim 1 , wherein a specific partition among the plurality of partitions includes a buffer storage portion for storing late-arriving log data.

3. The method of claim 1 , wherein applying the first log query to one or more partitions corresponding to the one or more time ranges comprises: Applying the query to each of the one or more partitions separately; as well as Return query results for each partition separately.

4. The method of claim 1 , further comprising: After executing the first log query: determining that the one or more partitions no longer meet the recency criterion; In response to determining that the one or more partitions no longer meet the recency criterion: removing the one or more partitions from the plurality of partitions, Wherein, removing the one or more partitions from the plurality of partitions results in an updated plurality of time ranges, the updated plurality of time ranges excluding a first time range associated with the first log query.

5. The method of claim 4, further comprising: After removing the one or more partitions from the plurality of partitions, performing a third log query by at least the following operations: determining a third time range associated with the third log query, wherein the third time range is within the first time range; in response to determining that a third time range is not within the updated plurality of time ranges; A third log query is applied to the historical data repository.

6. The method of claim 1 , wherein the log data is received in a raw format, and wherein storing the log data in a historical data repository comprises: parsing log data in an original format to obtain log data in a parsed format, wherein the parsed format includes one or more searchable parameters; Log data is stored in the historical data store in both raw and parsed formats.

7. The method of claim 1 , further comprising: receiving a third time range associated with the first log query that is different from the first time range; identifying a delta between the first time range and the third time range as a fourth time range; as well as The first log query is applied to one or more partitions of the plurality of partitions corresponding to a fourth time range.

8. A method for implementing a log parser in a log analysis system, comprising: receiving first log data; Storing the first log data in an indexed data repository comprising a plurality of collections, wherein storing the first log data in the indexed data repository comprises: In response to at least (a) determining that a first time associated with the first log data is within a first time range associated with a first set of the plurality of sets, and (b) determining that the first set is currently open: storing the first log data in the first collection; determining whether a first value of a first size of log data placed into the first set is below a first size threshold; In response to at least determining that a first value of a first size of the log data placed into the first collection is not below a first size threshold: Close the first set; adding a second collection to the plurality of collections, wherein the second collection is associated with a second time range different than the first time range, and the second collection is currently open; After closing the first set, receiving second log data different from the first log data; Storing the second log data in the indexed data repository, wherein storing the second log data in the indexed data repository comprises: In response to at least (a) determining that a second time associated with the second log data is within a first time range associated with a first set in the plurality of sets, and (b) determining that the first set is not currently open: determining whether a second value of the first size of the log data placed into the first set is below a second size threshold that is greater than the first size threshold; In response to at least determining that a second value of the first size of the log data placed into the first collection is below a second size threshold that is greater than the first size threshold: storing the second log data in the first set; After closing the first set, receiving third log data, wherein the third log data is different from the second log data and different from the first log data; Storing third log data in the indexed data repository, wherein storing third log data in the indexed data repository comprises: In response to at least (a) determining that a third time associated with the third log data is within a first time range associated with a first set in the plurality of sets, and (b) determining that the first set is not currently open: determining whether a third value of the first size of the log data placed into the first set is below a second size threshold; In response to at least determining that a third value of the first size of the log data placed into the first collection is not below a second size threshold: adding a third collection to the plurality of collections; storing the third log data in the third set; The method is performed by at least one device including a hardware processor.

9. The method of claim 8, further comprising: In addition to storing the first log data in the indexed data repository, storing the first log data in a historical data repository; The indexed data repository and the historical data repository are separate.

10. The method of claim 8, further comprising: receiving fourth log data; Storing fourth log data in the indexed data repository, wherein storing fourth log data in the indexed data repository comprises one of the following: (i) in response to at least (a) determining that a fourth time associated with fourth log data is within a first time range associated with a third set of the plurality of sets, and (b) determining that the third set is currently open: determining whether a fourth value of the second size of the log data placed into the third set is below the first size threshold; In response to at least determining that a fourth value of the second size of the log data placed into the third set is below the first size threshold: storing the fourth log data in the third set; or (ii) in response to at least (a) determining that a fourth time associated with fourth log data is within a second time range associated with a second set of the plurality of sets, and (b) determining that the second set is currently open: determining whether a fifth value of the log data of the third size placed into the second set is below a first size threshold; In response to at least determining that a fifth value of the third size of the log data placed into the second set is below the first size threshold: The fourth log data is stored in the second set.

11. The method of claim 8, further comprising one or more of the following: Further in response to determining that a first value of a first size of the log data placed into the first collection is not below a first size threshold: setting an end time associated with the first collection based on a first time associated with the first log data; Further in response to determining that a second value of the first size of the log data placed into the first collection is below a second size threshold: An end time associated with the first set is updated based on a second time associated with the second log data.

12. The method of claim 8, wherein the first size threshold comprises at least one of a threshold number of log entries and a threshold storage size.

13. The method of claim 8, further comprising: receiving a query associated with a specific time range, wherein a first time range is within the specific time range; The query is applied to the first set and the third set, but not to the second set.

14. One or more non-transitory machine-readable media storing instructions that, when executed by one or more hardware processors, result in performance of the method of any one of claims 1-13.

15. A system for implementing a log parser in a log analysis system, comprising: at least one device comprising a hardware processor; The device is configured to perform the method according to any one of claims 1 to 13.

Images