Method and system for implementing a log parser in a log analysis system

Abstract

The present invention relates to methods and systems for implementing log parsers in log analysis systems. Systems, methods and computer program products are disclosed for implementing log analysis methods and systems that can configure, collect and analyze log records in an efficient manner. Improved methods have been described for automatically generating log parsers by analyzing the line content of logs. Furthermore, efficient methods have been described for extracting key-value content from log content.

Description

[0001] This application is a divisional application of the application patent application with application number 201680029404.0, application date 1 April 2016, and invention title "Method and system for implementing log parsers in a log analysis system". BACKGROUND

[0002] Many types of computing systems and applications generate large amounts of data related to or caused by the operation of that computing system or application. These large amounts of data are stored into collections such as log files / records, which can be reviewed at a later time period if there is a need to analyze the behavior or operation of the system or application.

[0003] Server administrators and application administrators can benefit by learning and analyzing the contents of system log records. However, collecting and analyzing these records can be a very challenging task. These challenges are for many reasons.

[0004] One significant problem involves the fact that many modern organizations have very large numbers of computing systems, each with a large number of applications running on those computing systems. Given the large number of disparate systems and applications running on these computing devices, it can be very difficult to configure, collect, and analyze log records in large systems. Moreover, some of these applications can actually run on and across multiple computing systems, making the task of coordinating log configuration and collection even more problematic.

[0005] Conventional log analysis tools provide basic capabilities for collecting and analyzing log records. However, when faced with the problem of large systems comprising large numbers of computing systems with large numbers of applications running on those systems, conventional systems do not scale efficiently. This is because conventional systems often work on a per-host basis, where whenever a new host is added or newly configured in a system, setup and configuration activities need to be performed, or even new log collection / configuration activities need to be performed for existing hosts. Given the large number of hosts in modern systems, this approach is very inefficient. Moreover, conventional approaches, especially on-premise solutions, also do not adequately allow for sharing of resources and analysis components. This results in a large and excessive amount of redundant processing and resource usage.

[0006] Conventional log analysis tools are also very inefficient when it comes to the construction of log parsers used by the log analysis tools. Log parsers are tools that understand how to parse entries within a log. Conventionally, log parsers must be manually constructed by people who must be both familiar with the exact format of the log files to be analyzed and skilled in the specific programming infrastructure that will be used to implement the parser.

[0007] One problem with the conventional approach of manually building a log parser is that the process requires significant time and resources from skilled technicians. Furthermore, this approach demands excessive manual resources to maintain the parser should the log file format change. Additionally, this manual method inevitably requires prior knowledge of the log file format.

[0008] Therefore, improved methods are needed to implement log analysis systems. More efficient methods are also required to implement log parsers for log analysis systems. Summary of the Invention

[0009] Some embodiments of the present invention address the aforementioned problems by providing a method for automatically building a log parser. Instead of requiring manual creation of the log parser content, the log content itself is used to build the parser.

[0010] According to some embodiments, a method, system, or computer-readable medium is provided that constructs a log parser by: identifying a log to be analyzed; creating a mapping structure that maps the contents of the log to identified element types for one or more data portions within the log; selecting data portions from the log; analyzing the data portions relative to the mapping structure to identify mutable and immutable portions; assigning at least one mutable portion to a data type with the fewest restrictions on the variability of values ​​detected in the at least one mutable portion; and automatically generating a regular expression for the log parser. In some embodiments, the regular expression may include immutable portions and placeholders for mutable portions to implement the log parser, wherein at least two different placeholders are associated with different data types.

[0011] In some embodiments, the inventive method for identifying variable and immutable parts can be performed by: identifying lines from the log for comparison with a mapping structure, starting from the beginning of the line and moving forward until a mismatch is identified, finding the next common character, marking the intermediate range as variable, and looping until the end of the line is reached.

[0012] Within the mapping structure, the element type can include at least one of string type, integer type, alphabetic character type, or field rule type, wherein the field rule type corresponds to the sequence of elements defined by the rule.

[0013] Multiple rows are grouped together as a single entry for analysis of the mapping structure. Alternatively, the contents of multiple rows can be manipulated into a single row.

[0014] The following steps can be used to identify delimiters for the analysis scope within a log: identify common elements within two lines of the log, score the common elements, score the common elements by considering their position in conjunction with one or more weighting factors, and select a common element as the delimiter based on the scoring results. In some cases, weighting factors may include rules corresponding to combinations of multiple elements. Furthermore, sums or averages can be calculated for the positions of common elements.

[0015] Key and value fields can be extracted from logs by: identifying a first key-value delimiter and iteratively identifying key-value pair delimiters within a line to determine the range for evaluating one or more key-value pairs; and iteratively traversing the line to extract key fields from the left side of the instance of the key-value delimiter and value fields from the right side. Preprocessing can be applied to the logs to categorize the field and value portions. In addition to or as an alternative to preprocessing, postprocessing can be applied to correct problematic assignments of content to key or value fields.

[0016] According to some embodiments, log parsers are employed in log analysis systems implemented as cloud-based and / or SaaS (Software as a Service) architectures. The raw log data processed by the log analysis system can originate from any log-generating source, such as a database management system (DBMS), database application (DB App), middleware, operating system, hardware component, or any other log-generating application, component, or system. Log monitoring can be configured using a configuration mechanism that includes user-operable UI controls for selecting and configuring log collection configurations and target representations. Log collection configurations include a set of information (e.g., log rules, log source information, and log type information) that identifies what data to collect (e.g., which log files), the location of the data to be collected (e.g., directory location), how to access the data (e.g., the log format and / or specific fields to retrieve within the log), and / or when to collect the data (e.g., periodically). Target representations identify "targets," which are individual components that contain logs and / or generate logs. These targets are associated with specific components / hosts in the customer's environment. The current embodiment provides a unique advantage to the invention by enabling the configuration of log collection / monitoring by associating targets with log rules and / or log sources. This is because users configuring log monitoring do not need to know exactly how logs for a given application are located or distributed across different hosts and components within the environment. Instead, users only need to select the specific target (e.g., the application) to be monitored and then configure the specific parameters under which the log collection process will be performed.

[0017] Other additional objects, features, and advantages of the invention are described in the detailed description, drawings, and claims. Attached Figure Description

[0018] Various embodiments are described below with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and elements with similar structures or functions are indicated by similar reference numerals throughout. It should also be noted that the drawings are intended only to facilitate the description of the embodiments. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention.

[0019] Figure 1A Example systems that can be employed in some embodiments of the present invention are shown.

[0020] Figure 1B A flowchart of a method that can be employed in some embodiments of the present invention is shown.

[0021] Figure 2 The report UI is shown.

[0022] Figures 3A-3C It provides detailed diagrams of the internal structure of the log analysis system and the components that interact with the log analysis system within the customer's environment.

[0023] Figures 4A-4C The method for implementing log collection configuration is shown.

[0024] Figure 5 A flowchart is shown illustrating a method for configuring log collection by associating log rules with targets.

[0025] Figure 6 A flowchart is shown illustrating a method for configuring log collection by associating log sources with targets.

[0026] Figure 7 A flowchart is shown illustrating a method for implementing a target-based configuration for log monitoring.

[0027] Figure 8 A more detailed flowchart is shown below illustrating a method for implementing target-based configuration for log monitoring according to some embodiments of the present invention.

[0028] Figure 9 Example XML configuration content is shown according to some embodiments of the present invention.

[0029] Figure 10 This shows the server-side information that will be included in the configuration file to facilitate log parsing.

[0030] Figure 11 A flowchart is shown as one possible method for implementing this aspect of some embodiments of the present invention.

[0031] Figure 12 An architecture for some embodiments of an inventive method for associating log analysis rules with variable locations is shown.

[0032] Figure 13 The extraction of additional data that is inconsistent across all log entries is shown.

[0033] Figure 14 Some example field definitions are shown.

[0034] Figure 15 A high-level flowchart of a method for implementing a log parser according to some embodiments of the present invention is shown.

[0035] Figure 16 A more detailed flowchart of a method for implementing a log parser according to some embodiments is shown.

[0036] Figures 17-1 to 17-21 A diagram illustrating the process of building a log parser is provided.

[0037] Figure 18 The process flow for resolving non-standard line formats is shown in an embodiment.

[0038] Figure 19 This demonstrates the manipulation or categorization of row content.

[0039] Figure 20 A flowchart is shown below illustrating a method for efficiently identifying correct delimiter elements within a set of log content, according to some embodiments of the present invention.

[0040] Figures 21-1 to 21-5 The process of delimiter recognition is illustrated.

[0041] Figure 22 Some example weights that can be applied to common elements in some applications of this invention are shown.

[0042] Figure 23 A flowchart is shown for an example method of performing key-value extraction.

[0043] Figures 24-1 to 24-12 The key-value extraction process is illustrated.

[0044] Figures 25-1 to 25-2 and Figures 26-1 to 26-2 Example row configuration is shown.

[0045] Figure 27 The architecture of an example computing system in which the present invention can be implemented is shown. Detailed Implementation

[0046] As noted above, many types of computing systems and applications generate large amounts of data related to or caused by the operation of that computing system or application. This large amount of data is then stored in collection locations such as log files / records, which can be reviewed at a later time if the behavior or operation of the system or application needs to be analyzed.

[0047] Some embodiments of the present invention provide a method for automatically building a log parser. Instead of requiring manual creation of the log parser content, the log content itself is used to build the parser. Further objects, features, and advantages of the invention are described in the detailed description, drawings, and claims.

[0048] While the following description illustrates the invention with respect to "log" data, the scope of the invention is not limited to the analysis of log data, and in fact, the invention can be applied to a wide range of data types. Therefore, the application of the invention is not limited to log data unless so specifically claimed. Furthermore, the following description may also interchangeably refer to the data being processed as a "record" or a "message," without intending to limit the scope of the invention to any particular format of data.

[0049] Log analysis system

[0050] This part of the disclosure provides a description of methods and systems for implementing large-scale log collection and analysis, which can be used in conjunction with log parsers constructed as described below.

[0051] Figure 1A An example system 100 for configuring, collecting, and analyzing log data is illustrated according to some embodiments of the present invention. System 100 includes a log analysis system 101, which in some embodiments is implemented as a cloud-based architecture and / or a SaaS (Software as a Service) architecture. This means that the log analysis system 101 can provide log analysis functionality as a service on a hosted platform, so that each customer requiring the service does not need to separately install and configure service components on their own network. The log analysis system 101 can provide log analysis services to multiple individual customers and can be scaled to serve any number of customers.

[0052] Each customer network 104 may include any number of hosts 109. Host 109 is a computing platform within the customer network 104 that generates log data as one or more log files. The raw log data generated within host 109 can originate from any log-generating source. For example, raw log data can originate from a database management system (DBMS), database application (DB App), middleware, operating system, hardware component, or any other log-generating application, component, or system. One or more gateways 108 are provided in each customer network to communicate with the log analysis system 101.

[0053] System 100 may include one or more users located at one or more user stations 103 who use system 100 to operate and interact with log analysis system 101. User stations include any type of computing station that can be used with or interface to log analysis system 101 within operating system 100. Examples of such user stations include, for example, workstations, personal computers, mobile devices, or remote computing terminals. User stations include display devices, such as monitors, for displaying a user interface to users located at the user station. User stations also include one or more input devices that allow users to provide operational control over activities of system 100, such as a mouse or keyboard for manipulating pointing objects in a graphical user interface to generate user input. In some embodiments, user station 103 may (but does not need to) be located within client network 104.

[0054] Log analysis system 101 includes user-accessible functionality at user station 101. For example, log analysis system 101 is implemented as a collection of engines, mechanisms, and / or modules (whether hardware, software, or a hybrid of hardware and software) to perform the configuration, collection, and analysis of log data. A user interface (UI) mechanism generates the UI to display classification and analysis results and to allow users to interact with the log analysis system.

[0055] Figure 1B A flowchart is shown illustrating a method for configuring, collecting, and analyzing log data using System 100. Figure 1B This discussion will refer to the... Figure 1A The components shown in system 100.

[0056] At 120, log monitoring is configured within the system. This can be done, for example, by the user / customer, to configure the type of log monitoring / data collection desired by the user / customer. Within system 101, a configuration mechanism 129, including UI controls, is operable by the user to select and configure the log collection configuration 111 and target representation 113 for log collection configuration.

[0057] As discussed in more detail below, the log collection configuration 111 includes a set of information (e.g., log rules, log source information, and log type information) that identifies what data to collect (e.g., which log files), the location of the data to be collected (e.g., directory location), how to access the data (e.g., the log format and / or the specific fields to retrieve within the log), and / or when to collect the data (e.g., periodically). The log collection configuration 111 may include out-of-the-box rules included by the service provider. The log collection configuration 111 may also include customer-defined / customer-customized rules.

[0058] The term "target" (or "target") is defined as a separate component within a customer's environment that contains logs and / or generates logs. These targets are associated with specific components / hosts within the customer's environment. An example target could be a specific database application associated with one or more logs or one or more hosts.

[0059] The current embodiment provides a unique advantage to the invention by enabling the configuration of log collection / monitoring by associating targets with log rules and / or log sources. This is because users configuring log monitoring do not need to know exactly how logs for a given application are located or distributed across different hosts and components within the environment. Instead, users only need to select the specific target (e.g., the application) to be monitored and then configure the specific parameters under which the log collection process will be performed.

[0060] This addresses a significant problem with conventional systems that require configuring log monitoring on a per-host basis, where setup and configuration activities are performed every time a new host is added or configured in the system, or even new log collection / configuration activities are needed for existing hosts. Unlike conventional methods, log analysis users can isolate themselves from the details of the exact host / component associated with the logs for a given objective. This information can be encapsulated in underlying metadata maintained by the system administrator, who understands the mappings between applications, hosts, and components within the system.

[0061] The next action at 122 is to capture log data according to the user configuration. The association between log rule 111 and the target representation is sent to the client network 104 for processing. An agent of the log analysis system exists on each host in host 109 to collect data from the appropriate logs on host 109.

[0062] In some embodiments, data masking can be performed on the captured data. Masking is performed at the time of collection, protecting customer data before it leaves the customer's network. For example, various types of information in collected log data (such as usernames and other personal information) may be sensitive enough to be masked before being sent to a server. For such data identification patterns, this data can be removed and / or replaced with proxy data before it is collected for the server. This allows the data to still be used for analytical purposes while hiding sensitive data. Some embodiments permanently remove sensitive data (e.g., change all such data to "***" symbols) or replace it with data that can be mapped so that the original data can be recovered.

[0063] At point 124, the collected log data is delivered from customer network 104 to log analysis system 101. Multiple hosts 109 in customer network 104 provide the collected data to a smaller number of gateways among one or more gateways 108, which then send the log data to edge service 106 at log analysis system 101. Edge service 106 receives the data collected from one or more customer networks and places the data in an inbound data repository for further processing in log processing pipeline 107.

[0064] At 126, the log processing pipeline 107 performs a series of data processing and analysis operations on the collected log data, which will be described in more detail below. At 128, the processed data is then stored in data storage device 110. Computer-readable storage device 110 includes any combination of hardware and software that allows for rapid, ready access to data located at computer-readable storage device 110. For example, computer-readable storage device 110 can be implemented as a computer memory operable by an operating system. Data in computer-readable storage device 110 can also be implemented as database objects, cloud objects, and / or files in a file system. In some embodiments, the processed data is stored in both a text / indexed data repository 110a (e.g., as a SOLR cluster) and a raw / historical data repository 110b (e.g., as an HDFS cluster).

[0065] At point 130, the reporting mechanism / UI 115 can be used to perform reporting on the processed data. For example... Figure 2 As shown, the reporting UI 200 may include a log search facility 202, one or more control panels 204, and / or any suitable application 206 for analyzing / viewing processed log data. An example of such a reporting component will be described in more detail below.

[0066] At point 132, event management can be performed on the processed data. One or more alarm conditions can be configured within the log analysis system, so that when an alarm condition is detected, the event management mechanism 117 provides event / alarm notifications to a specified set of users.

[0067] At 134, the remediation action engine 119 can perform any necessary actions to be taken within the client network 104. For example, a log entry indicating a database system outage may be received. When such a log entry is identified, a possible automated remediation action is to attempt to restore the database system to operation. The client can create a remediation action script to address this situation. A trigger can be executed to run the script to perform the remediation action (e.g., the trigger causes instructions to be sent to an agent on the client network to run the script). In an alternative embodiment, an appropriate script for this situation is pushed down from the server to the client network for execution. Furthermore, at 136, any other additional functionality and / or actions may be appropriately taken, at least based on the processed data.

[0068] Figure 3A A more detailed diagram is provided of the internal structure of the log analysis system in the host environment 340 and the components within the client environment 342 that interact with the log analysis system. This architecture 300 is configured to provide a process for log monitoring capable of processing large volumes of log data digests.

[0069] Within a client environment 342 on a single client host / server 344, the LA (Log Analysis) agent 333 obtains log monitoring configuration data 332 (e.g., sniffer configuration or target-side configuration information) and invokes a log file 336 sniffer (also referred to herein as a "log collector") to collect log data from one or more log files 338. A daemon manager 334 can be used to interface with the log file sniffer 336. The log file sniffer 336 reads from one or more log files 338 on the host machine 344. The daemon manager 334 retrieves the log content and packages it so that it can be returned to the LA agent 333. It should be noted that the system can include any number of different types of sniffers, and the log sniffer 336 is merely an example of a single type of sniffer that can be used in the system. Therefore, other types of sniffers, such as sniffers for monitoring the registry, databases, window event logs, etc., can be employed in various embodiments of the invention. Furthermore, in some embodiments, the log sniffer is configured to process collection files / compressed files, such as Zip files.

[0070] LA agent 333 sends the collected log data to gateway agent 330. Gateway agent 330 packages the log data collected from multiple client hosts / servers, thus essentially acting as an aggregator of log content from multiple hosts. The packaged content is then sent from gateway agent 330 to edge service 306. Edge service 306 receives large amounts of data from multiple gateway agents 330 from any number of different client environments 342.

[0071] Given the potentially large volume of data that can be received at edge service 306, the data is immediately stored in inbound data storage device 304 (“Platform Inbound Repository”). This acts as a queue for the log processing pipeline 308. Data structures are provided to manage items to be processed within the inbound data repository. In some embodiments, a messaging platform 302 (e.g., implemented using a Kafka product) can be used to track items pending processing within the queue. Within the log processing pipeline 308, queue consumer 310 identifies the next item to be processed in the queue, which is then retrieved from the Platform Inbound Repository. Queue consumer 310 includes any entity capable of processing work leaving the queue within the system, such as processes, threads, nodes, or tasks.

[0072] The retrieved log data goes through the "parsing" phase 312, in which log entries are parsed and broken down into specific fields. As discussed in more detail below, the "log type" configured for the log specifies how log entries are broken down into the desired fields.

[0073] In the "Normalization" phase 314, the identified fields are normalized. For example, the "Time" field can be represented in any number of different ways across different logs. This time field can be normalized to a single recognizable format (e.g., UTC format). As another example, the word "error" can be represented in different ways across different systems (e.g., all uppercase "ERROR", all lowercase "error", initial capital "Error", or the abbreviation "err"). This situation may require normalizing the different word forms / types to a single format (e.g., the non-abbreviated term "error" in all lowercase).

[0074] The “Transformation” phase 316 can be used to synthesize new content from log data. As an example, and discussed in more detail below, “tags” can be added to the log data to provide additional information about log entries. As another example, field extraction can be performed to extract additional fields from existing log entry fields.

[0075] The "Condition Evaluation" phase 318 is used to evaluate the log data against specified conditions. This phase can be executed to identify patterns within the log data and to create / identify alert conditions within the log. Any type of notification can be executed during this phase, including, for example, emails / text messages / calls sent to an administrator / customer or warnings sent to another system or mechanism.

[0076] The log writer 320 then writes the processed log data to one or more data repositories 324. In some embodiments, the processed data is stored within both a text / indexed data repository (e.g., as a SOLR cluster) and a raw and / or historical data repository (e.g., as an HDFS cluster). The log writer can also send the log data to another processing phase 322 and / or a downstream processing engine.

[0077] As Figure 3B shown, some embodiments provide a side-loading mechanism 350 to collect log data without proceeding through the client-side proxy 333. In this approach, the user logs into the server to select one or more files on the local system. The system will load the file at the server and sniff the file (e.g., by having the user provide the log type, attempting possible log types, rolling through different log types, or making an educated "guess" about the log type). The sniffing results are then passed to the edge service and the process as described above. In Figure 3C embodiments, only the side-loading mechanism 350 exists to collect log files – where the proxy / sniffer entity is not installed and / or not required on the client server 344.

[0078] Figures 4A-4B illustrates a method for implementing a log collection configuration. This method allows for very large-scale configuration of how to monitor log files with one or more log entries. In some embodiments, a log entry corresponds to a single logical line from the log file. In an actual log file, since a carriage return is part of the log entry content, a single entry can span multiple lines. This entire content is considered a single "entry". Each entry starts with "#<date" and can occupy a single physical line in the file or multiple lines separated by carriage returns.

[0079] In this model, "Log Type" 406 defines how the system reads log files and how it breaks down the log files into their parts. In some embodiments, log files contain several basic fields. The basic fields present may differ for different types of logs. A "Basic Parser" can be used to break log entries down into specified fields. The Basic Parser can also perform transformations. For example, date fields can be converted to a normalized format and time adjusted to UTC so that data from multiple locations can be mixed together.

[0080] A "log source" 404 defines where log files are located and how they are read. In some embodiments, a log source is a named definition that contains a list of log files described using patterns and the parser required to parse those files. For example, a source could be "SSH log files". This source could list each log file associated with SSH individually, or it could use wildcards (e.g., " / var / log / ssh*") to describe the log files. For each pattern, a base parser can be selected (e.g., by the user) to parse basic fields from the files. This approach can be used to ensure that all files conform to the same basic parsing structure for a single pattern. For a source, multiple log types can be selected, with priority given to those possible types. For example, types A, B, and C can be identified, with the parser working through each of these types to determine if the source matches one of these identified types. Therefore, for each pattern, the user can select multiple base parsers. In some embodiments, the same source can be matched against multiple types and can be analyzed using multiple types.

[0081] "Log rule" 402 defines a set of sources along with the conditions and actions to be triggered during continuous monitoring. "Target" 408 identifies individual components in the IT environment that contain logs. In some embodiments, rules are associated with targets to initiate the monitoring process.

[0082] exist Figure 4A In this embodiment, one or more log rules are associated with one or more targets. Figure 4B In an alternative embodiment, one or more log sources may be associated with one or more targets to create instances of the target. Figure 4C In some embodiments, log rules are not even provided as a method for creating associations – only associations from log sources to targets are provided to create target instances. Each of these methods will be described in more detail below.

[0083] Figure 5A flowchart illustrating a method for configuring log collection by associating log rules with targets is shown. At 502, one or more log rules are created. The rules are processed by the rule engine within the log processing system to achieve rule-based processing of a given target. Therefore, each rule will include specific logic for processing the given target it is associated with.

[0084] In some embodiments, rules can be used to specify a target type that identifies the type of target the rule is intended to address. Rules can be specified for a single target type or multiple target types. For example, when monitoring log files used for a database instance, the target type can be set to "database instance" so that reporting of activity in the logs is targeted at the appropriate target type; in some embodiments, even if the rule can be configured for "file" as a log type, the target type can still be any managed target type, such as a database.

[0085] Rules can specify a source type, which identifies the type of log file the rule is intended to address. For example, a rule could specify the log file type as: (i) file: OS-level log file; (ii) database table: a table in a database that stores log content; (iii) Windows event log: reads events from Windows events as log content.

[0086] You can specify target attribute filters in the rules to filter targets and specify the conditions under which the rules apply, such as a specific operating system (OS), target version, and / or target platform. For example, a user can create rules that apply only to a given OS on a given platform (e.g., only to Linux OEL5 on x86_64 hardware).

[0087] In some embodiments, when creating a rule, the rule may also include: (a) the name of the rule; (b) a severity level that indicates how significant the result of the rule is if the rule causes an event to be generated; (c) a description of the rule; and / or (d) a textual reason for why such monitoring is being performed.

[0088] In some embodiments, one or more conditions can be established for a rule to be "triggered". Multiple conditions can be specified, each of which can be combined with other conditions using Boolean operators. For example, a set of conditions ORed with other conditions means that if any of these conditions matches an entry in the log file being evaluated, then that entry triggers the rule. When conditions are ANDed together, all clauses of the condition must be satisfied for the condition to trigger an entry in the log file. The specified action is then taken as a response to the matched entry. Here is an example conditional clause including a regular expression: "MESSAGE contains "START: telnet pid=[0-9]*from=[.]*", where this condition triggers the rule if the message matches the regular expression.

[0089] The “operators” in the conditions are used to perform comparisons. Here are some example operators that may be used in some embodiments of the invention: (a) <, >, >=, <=: compares values ​​that are greater than or less than (or equal to) a set value; (b) Contains: pattern matching with the ability to include regular expression clauses, where implicit wildcards can be placed at the beginning and end unless the user uses the ^ and $ regular expression symbols to specify the beginning or end of the string; (c) In: a list of possible values; (d) Is: an exact string match (no regular expression capability); (e) Is Not; (f) Does Not Contain; (g) Not In: a list of unmatched values.

[0090] Actions can be specified to identify what to do when a match is found on a selected source for a given condition. For example, one possible action is to capture the complete log entry as an observation when a rule's condition is matched. This approach allows the system / user to save the complete entry as an observation in a repository when monitoring logs from any source and seeing a single entry that matches the rule's condition.

[0091] Observations are stored for later viewing via the log observation UI or other reporting features. Another possible action is to create an event entry for each matching condition. This approach raises an event when a log entry is deemed to match a specified condition. In some embodiments, the event is created directly at the broker. The source definition defines any special fields (if any) that might be needed to capture the event. An additional option for this action is to bundle duplicate log entries at the broker and report the event at most once within a user-specified timeframe. Matching conditions can be used to help identify the presence of duplicate entries. Another example action is to create a metric for a rule to capture each occurrence of a matching condition. In this approach, a metrics subsystem is used to create a new metric for the rule. Subsequently, when a log entry exists that matches the condition of the rule, a certain number of fields are captured as metric data and uploaded as part of the metric. These fields can be selected to include information such as "key" fields (e.g., target, time, source, etc.).

[0092] At 504, one or more targets are identified in the system. The targets are individual components within the customer environment that contain logs. These targets are associated with specific components / hosts within the customer environment. Example targets include hosts, database applications, middleware applications, and / or other software applications associated with one or more logs or one or more hosts. Further details on how to specify targets are described below.

[0093] At point 506, associations are established between objectives and rules. Metadata can be maintained in the system to track the associations between a given objective and a given rule. A user interface can be provided that allows users to view which objectives a selected rule is associated with and / or add more associations, where association is the way a rule becomes active by associating it with a real objective.

[0094] Subsequently, at point 508, log collection and processing are performed, at least in part, based on the association between rules and targets. As discussed in more detail below, target-based configuration can involve various types of configuration data created on both the server and target sides to enable log collection and processing.

[0095] The current embodiment offers a unique advantage in configuring log collection / monitoring capabilities by associating targets with log rules. This is because users configuring log monitoring do not need to know exactly how logs for a given application are located or distributed across different hosts and components within the environment. Instead, users only need to select the specific target (e.g., the application) to be monitored and then configure the rules to be used to perform the log collection process.

[0096] This addresses a significant problem with conventional systems that require configuring log monitoring on a per-host basis, where setup and configuration activities are performed whenever a new host is added or configured in the system, or even new log collection / configuration activities are needed for existing hosts. Unlike conventional methods, log analysis users can isolate themselves from the details of the exact host / component associated with the logs for a given objective. This information can be encapsulated in underlying metadata maintained by the system administrator, who understands the mappings between applications, hosts, and components within the system.

[0097] As an alternative to or in addition to rules, log processing can also be configured by associating log sources with targets. Figure 6 A flowchart illustrating a method for configuring log collection by associating log sources with targets is shown. At 602, one or more log sources are created. A log source defines where log files are located and how they are read. Log sources can define source types that indicate how source content is collected. The following are example source types: (a) File – identifies a readable file from the OS level that can be accessed using regular OS-level file operations; (b) Database Table – a table that stores log entries (e.g., a database audit table); (c) Windows Event System – provides an API for accessing event records. One or more source names can be defined for log sources. Furthermore, log sources can be associated with a description of the source. Note that log sources can also be used when creating log monitoring rules (as described above).

[0098] Log sources can also be associated with file patterns and / or pathname expressions. For example, " / var / log / messages*" is an example of a file pattern (which may actually be associated with multiple files). One reason for using file patterns in current log analysis systems is that the exact location of the logs being monitored may change. Sometimes, the system expects the logs to be located in a specific place, such as a specific directory. When a system processes a large number of streaming logs, it may not know which directory the logs are expected to be in. This prevents systems that rely on static log file locations from functioning correctly. Therefore, file patterns are useful for addressing these potentially changing log locations.

[0099] In some embodiments, a log source is created by specifying a source name and description for the log source. The definition of a log source can include included filename patterns and excluded filename patterns. A filename pattern is a pattern corresponding to a file (or directory) to be included for the log source. An excluded filename pattern corresponds to a pattern of a file (or directory) explicitly excluded from the log source; this is useful, for example, if the included filename pattern identifies a directory with many files and some of those files (such as pseudo-files or non-log files) are excluded using the excluded filename pattern. For each pattern, the system captures the pattern string, description, and basic parser (log type) that will be used to parse the file. The basic parser can define the basic structure of the file, such as how to parse data, hostnames, and messages from the file.

[0100] The definition of a log source can also specify whether the source contains security log content. This is available so that the source creator can specify special roles that users must have to view any log data that may be captured. This log data can include security-related content that is not visible to any target owner.

[0101] As noted above, log rules can reference log sources and vice versa. In some embodiments, system metadata tracks these associations to maintain a count of rules currently using a source. This helps in understanding the impact if a source and / or rule is changed or deleted.

[0102] At 604, one or more targets are identified. As noted above, a target is a component within the environment that contains, corresponds to, and / or creates logs or other data to be processed, wherein the target is associated with a specific component / host in the customer environment. Example targets include hosts, database applications, middleware applications, and / or other software applications associated with one or more logs or hosts.

[0103] At position 606, an association is established between the target and the source. Metadata can be maintained in the system to track the association between a given target and a given source. A user interface can be provided that allows users to see what targets are associated with a selected source and / or add more associations.

[0104] At 608, the target-to-source association creates a concrete instance of the log source. For example, consider a log source that typically specifies a given file located at a given directory location (e.g., c: / log_directory / log_file). It's possible that any number of servers within a client environment (server A, server B, server C, server D) could have a copy of that file (log_file) in that directory (c: / logdirectory). However, by associating a concrete target (e.g., server A) with the log source, this creates an instance of the log source so that the new instance is specific to the log file in the specified directory on the concrete target (e.g., starting to monitor c: / log_directory / log_file specifically on server A).

[0105] Subsequently, at point 610, log collection and processing are performed at least in part based on the association between rules and log sources. As discussed in more detail below, target-based configuration can involve various types of configuration data created on both the server and target sides to enable log collection and processing activities.

[0106] There are many advantages to using this type of model for configuring log collection. One advantage is that log types, sources, and rules can be easily reused as needed. Furthermore, this approach avoids a large amount of repetitive configuration by enabling sharing across multiple levels. Additionally, users can create custom rules using sources and log types defined by others or provided with the product. This approach is also easy to build upon with shared knowledge.

[0107] Associating rules / sources with targets provides knowledge of where log collection is physically performed via the agent. This means users don't need to know anything about the location of the target. Furthermore, it facilitates a large number of associations between rules / sources and targets. In some embodiments, rules / sources can be automatically associated with all targets based on configuration. As noted above, service providers can offer out-of-the-box configurations. Additionally, users can create their own configurations, including extensions of the provided out-of-the-box configurations. This allows users to customize without building their own content.

[0108] Figure 7 A flowchart illustrating a method for implementing a goal-based configuration for log monitoring is shown. This process generates, deploys, and / or updates configuration profiles for log monitoring. In some embodiments, the configuration profile is implemented as a configuration file used by the log monitoring system to manage and implement the log monitoring process.

[0109] At point 700, goal-based processing is initiated. Example methods for initiating goal-based processing include, for instance, installing a log analysis agent onto a specific log collection location. Goal-based processing involves associating one or more goals with one or more log sources and / or rules.

[0110] At 702, configuration information is generated for target-based processing. In some embodiments, the target-based configuration file is implemented as a configuration XML file, but other formats can also be used. The target-based configuration file can be created at the main site (e.g., to create major version 704), and then the specific version is passed to the server side and the target side.

[0111] The target-side data 708 may include configuration details relevant to log collection operations. This includes, for example, information about log source and target details. The server-side data 706 may include configuration details relevant to server-side log processing. This includes, for example, information about parser details.

[0112] In some embodiments, the database at the server maintains a major version and a target version of configuration data. As noted above, the target version includes configuration details related to the log collection effort and is passed to the client environment for use by agents in the client environment to collect appropriate log data from the client environment. The major version includes the complete set of configuration details required at the server and becomes "server-side" data when selected and used for processing at the server. This can occur, for example, when log data collected at the target is passed to the server, where the transmission of log data includes an identifier that uniquely identifies the target-side data used to collect the log data (e.g., ...). Figure 9 The example target-side data shows the configuration version or "CV" number 903. When this data is received at the server, an identifier is used to determine the corresponding major version of the data with the same identifier number (e.g., as shown in the example target-side data). Figure 10 (As shown in field 1003 of the example server-side data). This primary version is then used as server-side data to process the received log data. Therefore, in this embodiment, primary version 704 and server-side data 706 are identical, but have different labels depending on whether the data is currently being used to process log data. In alternative embodiments, for example, where the data is used on multiple servers with different configuration details, the primary version may differ from the server version.

[0113] At 710, the configuration data is then distributed to the appropriate locations within the log processing system. In some embodiments, target-side data 708 serves as... Figure 3A The sniffer configuration file 332 shown is distributed to the client system. Regarding server-side data 706, the data is as follows:Figure 1A The log configuration file 111 shown is "distributed," where the distribution does not actually require distributing data across the network, but merely indicates that the data is obtained from another component within the server (e.g., as needed).

[0114] Next, at point 712, log collection processing is performed at the target location using the target-side configuration data. Additionally, at point 714, server-side log processing is performed using the server-side configuration data.

[0115] Figure 8 A more detailed flowchart of a method for implementing target-based configuration for log monitoring according to some embodiments of the present invention is shown. At 802, one or more work items for handling target associations are created in the system. For example, this type of work may be created when a log analysis agent is installed onto a target, wherein identification of such installation causes work items to be created for target-based configuration data. A list of target types having at least one automatic association rule (e.g., from an associated database) is identified. A list of targets that need to be associated with automatically enabled rules is generated. These steps are equivalent to a producer entity / process placing association tasks into a queue (e.g., a database table), and the association tasks are then processed by one or more consumer entities / processes.

[0116] One or more consumer / worker entities can periodically wake up to process work items. For example, a worker entity (e.g., a thread or process) wakes up (e.g., every 10 seconds) to check for any pending associated tasks. A collection of one or more workers will iterate through the tasks to process the work in the queue.

[0117] At step 804, one of the workers identifies the associated task to be processed. At step 806, the associated request is processed by accessing information collected for rules, sources, parsers, fields, and / or targets. This action identifies the target being addressed, locates the target, and then looks up details of the log sources and / or log rules already associated with that target.

[0118] At step 808, the worker then generates configuration content for the specific associated task it is processing. In some embodiments, the configuration content is implemented as XML content. This action creates both target-side details and server-side details for the configuration data. For the server side, this action creates configuration data for the server to process the collected log data. For example, it creates XML-formatted parser details for the server-side data to be used for the log data to be received. For the target side, this action creates configuration data for log collection from the target. For example, as discussed below, variable pathnames (e.g., with variables instead of absolute pathnames) can be specified for a given log source to identify the directory containing the log files to be monitored. At step 808, these variable pathnames can be replaced with the actual pathnames and inserted into the target-side data.

[0119] At 810, it is determined whether there are any additional related tasks to process. If there are additional tasks in the queue, the process returns to 804 to select another task to process. If not, the configuration information is finalized at 812.

[0120] It's important to note that the same configuration / XML file can be used to address multiple associations. For example, if multiple targets reside on the same host, a single configuration file can be generated for all targets on the host. In this case, step 808 described above appends the XML content to the same XML file using a loop for multiple iterations.

[0121] Updates can occur in a similar manner. When changes occur that require updating data, one or more new associated tasks can be queued and addressed as described above. Furthermore, unassociation can occur, for example, when the log analysis agent is unloaded. In this case, the configuration file can be deleted. When a target is deleted, a message can be broadcast to notify all listeners of the event via the target model service. This message can be consumed to delete the corresponding association and update the XML content.

[0122] Figure 9Example XML configuration content 900 is shown according to some embodiments of the present invention. This is an example of target-side content that can be placed on a host that holds the target. This XML configuration content 900 defines rules for collecting Linux system message logs with the file pattern " / var / log / messages*" on the host XYZ.us.oracle.com. Section 902 identifies the base parser used for the association being addressed. Section 903 provides an identifier for the version number ("configuration version" or "CV") of content 900, which is used to match corresponding server-side data with the same version number. Section 904 identifies the ID of the log rule. Section 906 identifies the specific target. Section 908 identifies the target type. Section 910 identifies the source type. Section 912 identifies the parser ID used for the source. The logs will be parsed based on a defined parser. These configuration files reside on a sniffer, and the log collection process collects logs based on defined log sources.

[0123] In server-side log processors, additional information can be included in configuration files to facilitate log analysis, such as... Figure 10 As shown in the server-side content section 1000. FieldDef section 1001 indicates the data type of the service. Log Source section 1002 indicates that the log has the type "os_file". BaseParse section 1004 defines the method for parsing log entries based on the regular expressions defined in section 1006. Section 1003 provides an identifier for the version number of content 1000, which is used to match corresponding target-side data with the same version number.

[0124] In addition to the automatic associations described above, target-source manual associations can also be performed. For example, a user interface can be provided to perform manual associations. This also results in the aforementioned actions being performed, but triggered by manual actions.

[0125] Resynchronization of target-source associations can be performed. To illustrate, consider that when a log analysis agent is installed, monitored targets connected through the agent can be associated with certain predefined log sources. Similarly, when the agent is uninstalled, these associations can be removed from the appropriate database tables. Furthermore, when a target is added for monitoring by the agent, it can be associated with certain predefined log sources for that target type, and when the target is removed from the agent, this association can be removed from the database tables.

[0126] Over time, these associations may become out of sync for various reasons. For example, when installing a log analysis agent, automatic associations may fail due to some network issue that causes configuration data to be lost during transmission. Furthermore, when targets are added or removed, events may not be handled correctly, thus preventing the configuration XML file from updating properly.

[0127] To handle these situations and maintain consistency in the association between targets and their corresponding log sources, a web service is provided in some embodiments to periodically synchronize associations. In at least one embodiment, only automatic associations are synchronized, while manually customized associations by the user are not synchronized.

[0128] Associations can be performed on specific log analysis agents. This action can be achieved by performing incremental analysis between targets in the data model data repository and targets in the log analysis data repository. Processing can occur in the following situations: (a) adding associations for targets that are in the data model data repository but not in the log analysis data repository; (b) deleting associations for targets that are not in the data model data repository but are in the log analysis data repository; (c) retaining the same associations for targets in both the data model data repository and the log analysis data repository, in the case of user customization. A potential problem with adding associations relates to the following situation: a user may have deleted all associations for a specific target, so there are no entries in the log analysis data repository, but entries exist in the data model data repository. The problem is that when the above methods are applied, unwanted automatic associations may be reintroduced after the synchronization operation. To avoid this situation, the system can log user actions to identify potential problems.

[0129] In addition, associations can be synchronized for specified tenants. When this action is performed, incremental analysis can be performed between the agent used for the data model data repository and the agent used for the log analysis data repository. The process can occur through the following: (a) adding associations for agents that are in the data model data repository but not in the log analysis data repository; (b) deleting associations for agents that are not in the data model data repository but are in the log analysis data repository; and (c) performing the same incremental analysis and synchronization as described above for agents in both the data model data repository and the log analysis data repository.

[0130] Synchronization can be performed on all tenant associations. When performing this action, agent-level synchronization as described for each tenant should be performed.

[0131] This document turns its attention to file-based models. One reason for using file-based models in log analysis systems is the potential for monitoring changes in the exact location of logs. Most of the time, a system expects logs to be in a specific location, within a specific directory. When a system processes large volumes of streaming logs, it may not be aware of which directory these logs are expected to reside in. This prevents systems that rely on static log file locations from operating correctly.

[0132] In some embodiments, the inventive method can associate log analysis rules with variable locations. One approach is to use metadata that replaces the variable portion corresponding to the location of the log file. A path expression is used to represent the pathname of the log file, where the path expression includes a fixed portion and a variable portion, with different values ​​implemented for the variable portion. The location placeholders are ultimately replaced with the actual location in the directory path.

[0133] Some implementations provide "parameters," which are flexible fields (e.g., text fields) that users can use to include or exclude filename patterns. Parameters can be implemented by enclosing their names in curly braces { and}. User-defined default values ​​are provided in this source. Users can then provide parameter overriding on a target-by-target basis when associating log monitoring rules using this source with targets. Overriding is particularly useful for changes to out-of-the-box content (e.g., overriding rules, definitions, etc., without actually changing the OOTB content). This is achieved, for example, by implementing a mapping / annotation table that includes user overriding and instructions for overriding OOTB content.

[0134] This is very helpful because, within the log source, you can define the paths to the log files you want to monitor. In some cases, the paths are fixed, such as in the Linux syslog file, where the path is " / var / log / messages*". However, in other cases, you might want to monitor database alert logs, where each database target will be mounted in a completely different path, and the path to find the alert logs might be different. For example, the alert logs for one database might be located at: / xxx / db / yyyy / oracle / diag / rdbms / set2 / set2 / The underlined portion of `alert / log*.xml` can differ for each database target. However, each target has the concept of target attributes. Included in these attributes are metadata that can be used to populate the variable portions of the path. In the current embodiment, this path can be alternatively represented as:

[0135] "{DIAGNOSTIC_DEST} / diag / rdbms / {SID} / {SID} / alert / log*.xml"

[0136] When this source is used in a rule and the rule is associated with a target, the system replaces the parameters "DIAGNOSTIC_DEST" and "SID" with parameters known to that target. This allows the system to associate a single rule and source with thousands of targets at once.

[0137] As another example, a user might want to monitor the pattern: " / xxx / oracle / log / *". In this case, " / xxx / oracle" is a variable path depending on the host. Alternatively, the pattern can be written as: "{INSTALL_DIR} / log / *". For this source, the user can provide a default value ( / xxx / oracle) to the INSTALL_DIR parameter. Later, when a rule is associated with a target, the user can provide the target overwrite value " / xxx / oracle" for this parameter on that target without creating a new source or rule.

[0138] For system-defined fixed parameters, there may be situations where users want to reference built-in parameters (e.g., ORACLE_HOME). In this case, the system will replace that variable with ORACLE_HOME, which is known to the selected target. The pattern can be written as: "{ORACLE_HOME} / log / *". This path will be automatically understood by the agent, where ORACLE_HOME is a special built-in parameter that does not require the user to set a default value. The system can provide a list of fixed parameters that integrators / users can choose to use.

[0139] Figure 11 A flowchart illustrating one possible method for implementing this aspect of some embodiments of the invention is shown. At 1102, location content for which variable location processing is desired is identified. This may be the case, for example, when the system is processing a large volume of streaming logs from potentially numerous and / or uncertain directory locations. The log data may reside at target locations addressed using pathnames that vary for different database targets.

[0140] At 1104, specify the path to the target location, which has a fixed portion and a variable portion. The variable portion can be represented by one or more parameters. During log processing, at 1106, replace these one or more parameters with values ​​corresponding to one or more target log files, where a single rule for implementing log monitoring is associated with multiple different targets to be monitored.

[0141] This approach is significantly more advantageous than methods where each log entry resides in a different directory that cannot be known in advance, and where a separate forwarder mechanism must be set up for each path. Instead, this approach can be used to set up a single rule for a very large number of paths.

[0142] In some embodiments, configuration information from the log analysis system can be coupled to this approach to configure and set rules for identifying log file assignments. Some examples of configuration information that can be used include, for example, how the database is connected, how components are connected, which data center is used, and so on.

[0143] Some embodiments specify how to map sources to targets based on the relationship between sources and targets. For example, a defined source Source1 can be assigned to all relevant targets belonging to a certain system. Any association type and / or rule can be used in this embodiment, where a common set of association types is used to provide configuration information useful for determining rules for log locations. Such association types can include, for example, "contains", "application_contains", "app_composite_contains", "authenticated_by", "composite_contains(abstract)", "cluster_contains", "connects_through", "contains(abstract)", "depends_on(abstract)", "deployed_on", "exposes", "hosted_by", "installed_at", "managed_by", "monitored_by", "provided_by", "runs_on(abstract)", "stores_on", "stores_on_db", and "uses(abstract)".

[0144] It's important to note that target relationship information / models can also be used in other ways. For example, the target model can be used to help correlate log entries to aid in root cause analysis. As another example, the host model can be used to compare all hosts within a system. For instance, if multiple databases exist in the first system, this feature can be used to view logs across these systems together, isolated from the database used in the second system.

[0145] Figure 12An architecture for some embodiments of an inventive method for associating log analysis rules with variable locations is shown. Here, the log analysis engine 1202 operates by accessing a log collection profile 1211. The log collection profile 1211 is implemented to represent a path in which a target location can have both a fixed portion and a variable portion. The variable portion can be represented by one or more location parameters. In this example, different locations can exist for logs 1202a, 1201b, and 1201c. By replacing the variable portion, the specific location of the log of interest can be selected by the log analysis engine 1202 and processed to generate analysis results 1213.

[0146] Here, reference 1210 can be accessed to identify the correct alternatives to the variable portions of the path used for the target location. Any suitable type of reference can be implemented. As mentioned above, the defined source Source1 can be assigned to all relevant targets belonging to a system, and / or association types and / or rules can also be used. Furthermore, target relationship information / models and references can be employed.

[0147] Therefore, embodiments of the present invention provide improved functionality for performing target-based log monitoring. Two possible use cases for this functionality include log monitoring and self-organizing log browsing. Log monitoring involves, for example, situations where there is continuous monitoring and capture of logs. Some embodiments of log monitoring involve some or all of the following: (a) monitoring any logs for any target and capturing important entries from the logs; (b) creating events based on some log entries; (c) identifying the existence of log entries that can affect compliance scores; (d) performing user-defined monitoring as well as integrator-defined monitoring; (e) capturing log entries that are not events to enable analysis on a subset of all logs; (f) use cases such as intrusion detection, potential security risk detection, and problem detection; (g) achieving long-term persistent storage of log content; (h) searching log content; and (i) customizable search-based views.

[0148] (j) Log anomaly detection and scoring.

[0149] Self-organizing log browsing involves scenarios where, for example, there is no continuous monitoring of logs. In this approach, users can browse live logs on a host machine without having to collect logs and send them up to a SaaS server. The model for configuring what to monitor is similar to the one described earlier. The difference lies in the fact that users can select rules, sources, and some filters from the UI, and then the search is sent down to an agent to retrieve matching log files and bring them back, storing them in temporary storage on the server. Users can continue to narrow their search on this result set. If the user adds another target, rule, or expands the time range, the system returns to the agent to retrieve only the incremental content, rather than retrieving the entire content again. Therefore, users can obtain the same benefits of log analysis without configuring continuous log monitoring. This feature can have very low latency because when the search is expanded, the system only needs to return to retrieve more data from the agent. All searches that narrow the current result set are compared with previously retrieved and cached data from the agent.

[0150] Embodiments of this invention can be used to store log data in a long-term centralized location within a raw / historical data repository. For example, a target owner in a company's IT department can monitor incoming issues for all targets under their responsibility. This could include thousands of targets (hosts, databases, middleware, and applications) managed by the company's SaaS log analytics system. Numerous log entries (e.g., hundreds of gigabytes) can be generated daily. For compliance reasons, these logs may need to be stored permanently, and based on these logs, data center managers may want a long-term overview, and IT administrators may want to search them to identify potential causes of specific problems. In this scenario, a very large volume of logs can be stored in a centralized storage device on which users can search and view log trends with acceptable performance. In some embodiments, log data can be stored in an offline storage repository. This can be used, for example, when data is kept online for a period and then transferred offline. This is particularly useful when different pricing tiers exist for different types of storage devices (e.g., offline storage is cheaper) and users can choose where their data is stored. In this approach, data can be kept offline and can be brought back online at a later point in time.

[0151] Logs can be searched to analyze potential causes of problems. For example, when a specific problem occurs for a target, the target owner can analyze logs from various sources to pinpoint the cause. Specifically, time-related logs from different parts of the same application or from different but related applications can be reported in an integrated view in a time-staggered format to help the target owner identify potential causes of the problem. The target owner can perform self-organizing searches to find identical or similar log entries over time, jump to log entries of interest, and then drill down to detailed messages and browse other logs generated before / after the point of interest.

[0152] In some embodiments, restrictions can be applied so that users can only access logs to which they have been granted access privileges. Different user tiers can be associated with access to different sets of logs. Various roles can be associated with permissions to access certain logs.

[0153] Several implementation examples can be used to view long-term log distribution, trends, and correlations. Because many logs are generated over long periods from many different targets and log sources, data center managers may want to view long-term log distribution and patterns.

[0154] Several implementations can be used to search logs to identify the cause of application outages. Consider a scenario where the target owner or IT administrator of a web application receives notifications from customers using the application reporting that they are unable to complete their online transactions and that a confirmation page does not appear after clicking the submit button. Using embodiments of the invention, the IT administrator can search application-generated logs by username and within the time frame of the problem report. The logs may contain application anomalies indicating a database error occurred when the application attempted to submit a transaction. By adding databases and their corresponding hosting servers to the search via log sources related to target relationships and their availability, the IT administrator can browse the logs around the time of the application anomaly to look for database errors, such as those related to partial disk failures on a hosting server and a large number of transaction submissions.

[0155] Several implementations can be used to view long-term log distribution, trends, and correlations through tags. Data center managers can define tags for logs collected in the data center, such as security logs for production databases, security logs for development servers, logs for test servers, noise logs, and so on. Data managers may be interested in understanding, for example, the distribution of logs by these tags over the past six months, their daily inbound rates during the last month, and whether there is any correlation between security log entries for production databases and changes in their compliance scores over a given time period.

[0156] Some implementations allow log data to be stored as metrics. In some implementations, the system stores several log fields as key fields. Key fields will include (but are not limited to): Time, Target, Rule, Source, and Log File. The system may also create hashes or GUIDs to distinguish possible log entries with the same time and all other key fields. When a rule for the metric action on a log entry is associated with a first target, a metric extension is created and deployed. This metric extension will be named similarly to the rule for ease of reference by the user.

[0157] In some embodiments, log monitoring rules have possible actions for creating an event when a log entry matches a condition of the rule. Furthermore, users will be able to instruct that this event should also trigger a compliance violation, which will affect compliance scores against compliance standards and frameworks.

[0158] As noted above, one possible use case is, for example, providing a log browser that is used to browse live logs on a host without collecting the logs and sending them to a SaaS server. Users could select rules, sources, and certain filters from the UI, and then the search would be sent down to an agent to retrieve matching log files and bring them back, storing them in temporary storage on the server. One use case for this feature is to allow users to browse short periods of log files across multiple targets in a system to try to discover the source of a problem, especially in customer environments with rich topology and dependency mappings. This content can be used to help find related elements and to display the logs together. This allows users to view, for example, the logs of all targets related to a given system and see what happened across all targets in chronological order. In many cases, when a target fails, it may be a dependent target experiencing the problem, rather than the target that is failing.

[0159] Users can choose to start a new log browsing session within the context of a system / group / individual target. If accessed from the target homepage, the target homepage context is retained. This means the page's shell still belongs to the target homepage, and only the content panel will contain the browsing UI functionality. This means the browsing UI can be implemented modularly to be dynamically inserted into other pages. In some embodiments, multiple lines of content can be provided for each entry to show additional details for each line. This is one line at a time, or the user can decide to perform this operation on all lines. Sorting of the parsed fields can be provided, but in addition, sorting can be used to view additional details for each line (including the original log entry).

[0160] Search filters can be provided. For example, search filters can be provided in the form of date ranges, such as Most Recent and Specific Date Range. Using the Most Recent option, users can enter a time and a timescale in minutes or hours. Using the Specific Date Range option, users will enter the start and end times. The date range options allow users to specify Targets, Sources, and Filters. These allow users to select what they want to see in this log browsing session. After users select targets, sources, and apply any filters, they can begin browsing the session to initiate a retrieval of logs from various targets, which will then be displayed on the interface.

[0161] Search queries can be implemented in any suitable manner. In some embodiments, natural language search processing is performed to implement search queries. Search processing can be used to perform searches across dependency graphs. Various relationships can be queried in the data, such as "runs on", "use by", "uses", and "memberof".

[0162] In some embodiments, the search query is a text expression (e.g., based on the Lucene query language). Users can enter a search query in the search box to search the log. Here are examples of what can be included in a search query: (a) items; (b) fields; (c) item modifiers; (d) wildcard search; (e) fuzzy search; (f) proximity search;

[0163] (f) Range search; (g) Boost item; (h) Boolean operators; (i) Grouping; (j) Field grouping; (k) Escape special characters.

[0164] A table view of search results can be provided. Some query refinement can be performed via table cells, allowing users to add / remove field-based conditions in the query text contained in the search box via UI actions. For example, when a user right-clicks a field, a pop-up window provides options for adding or removing conditions that filter logs during the search. This is convenient for users to modify the query text, and using this method, users can refine their queries at the field level without needing to know the internal field names.

[0165] There are many ways to list fields in a search results table for users to select / deselect for display purposes. One example method is based on static metadata, and another possible method is based on dynamic search results.

[0166] For list fields based on static metadata, use the basic field shuttle to list all defined fields. Some example fields that can be defined by log entry metadata include: (a) log file; (b) entry content; (c) rule name; (d) source name; (e) resolver name; (f) source type; (g) target type; and (h) target name. The values ​​of these fields can be obtained from the broker, while the log entry (although the source, resolver, rule, and target are all GUIDs / IDs) will need to be looked up when displayed.

[0167] For list fields based on dynamic search results, the first n (e.g., 10) fields will be displayed, which will be suggested as the most important for the search. A "More Fields" link will trigger a pop-up window allowing the user to select additional fields. The user can see more information about those fields in the pop-up window than from the View menu. When listing fields, the system can use any suitable algorithm, such as assigning numbers to each field influenced by factors like how many rows in the search results have non-empty values ​​or how many distinct values ​​exist across all search results for that field.

[0168] Given the numerous dynamic fields available for users to select / deselect, it's desirable that users be able to save their field selections (field names and sizes). The system can store the last selected field so that when the user returns to the page, he / she still receives the field he / she last selected.

[0169] There may be a very large number (e.g., thousands) of log entries generated by a search, and a user may not be able to browse all of them to find the log entry of interest. For a specific search, the user should be able to drill down to the details of the search results with a few clicks. In some embodiments, features include clickable bar charts and table pagination. Using these navigation features, along with customizable time ranges, users should be able to quickly jump to a point of interest. Accordingly, some embodiments provide drilling down to higher levels of detail, so users can easily navigate to the desired log entry via bar charts. An example use case is that after a user drills down several levels, they may want to drill back up to a previous level to go down another bar. After a user identifies a log entry of interest through some searches, they may want to explore logs from specific log sources surrounding the log entry of interest, or explore logs from multiple log sources surrounding the log entry of interest in a time-interleaved pattern. Some embodiments provide the option to allow the user to browse forward / backward page by page of logs surrounding a specified log entry. A graphical view of the search results can be provided. This allows the user to select fields to graphically render the results.

[0170] Some embodiments involve improved techniques for addressing log distribution, trends, and relevance. For search results from a specific search, the distribution can be based on log counts to provide the user with some high-level information about the logs. For each distribution type, the top n (e.g., 5 or 10) items and the number of logs found are listed (where a "More..." link will lead to a pop-up listing all other items). When a user selects a specific item, only the logs corresponding to that item will be displayed in the table on the right, thus this action is equivalent to filtering the search results by that item. This information can be presented in the following ways: (a) by target type; (b) by target, such as target owner and / or lifecycle status; (c) by log source; (d) by tag. In addition to displaying search results in the results table, the system can also provide the user with the option to switch between a table view and the corresponding distribution graph view.

[0171] In some embodiments, results can be filtered by selecting distribution items. Users can filter the results table by selecting one or more distribution items. By default, all distribution items are selected and all log entries are listed in the results table. After selecting one or more distribution items, users can navigate through the log entries via pagination. When one or more distribution items are selected, the selection of distribution items will be reset to all distribution items being selected when the user clicks the search button to perform a new search.

[0172] Some embodiments provide features that illustrate trends in search results. Some embodiments provide features that illustrate the relevance of search results. Regarding this feature, some embodiments provide start-up links that allow users to navigate to search / view detailed logs when performing correlation analysis between events, metrics, and infrastructure changes. Start-up links may be provided, for example, to guide users to an IT analytics product to analyze / view detailed events / metrics when they want to see some more general information related to the logs here.

[0173] Another feature in some embodiments involves processing time-extended field definitions. Even when using the same baseline log type, individual log entries may contain inconsistent information from one log to the next. In some embodiments, this can be addressed by defining basic fields common to the log type, which then allow for extended field definitions for additional data in the log entries.

[0174] To illustrate, consider the source definition that defines the log files to be monitored. The log files are parsed into their basic fields based on the log type definition. Additional data inconsistent across all log entries can be extracted, such as... Figure 13 As shown in diagram 1300. In this diagram, the basic fields parsed from the log entry are Month, Day, Hour, Minute, Second, Host, Service, Port (optional), and Message. The goal is to extract the IP address and port from the second log entry. For example, since not every log entry has this structure, this goal may not be available in some implementations where it is part of the log type. Here, the Message field for the second entry has the following:

[0175] Accepted publickey for scmadm from xxx.xxx.1.1port xyz ssh2 In some embodiments, extended field definitions on the Message field are defined using a format such as the following:

[0176] For this log entry, the accepted publickey for `.*from{IP Address}port{Port}ssh2` will resolve two new fields: IP Address and Port, which will then be available for reporting, searching, etc. This extraction occurs when the data is processed during collection.

[0177] According to some embodiments, the process for defining the field for implementing processing time extension includes: identifying one or more log files to be monitored, wherein some entries in the one or more log files may include additional data that is not present in other entries or is inconsistent with entries in other entries, such as an additional IP address field in one entry that does not appear in another entry; identifying source definitions for the one or more log files to be monitored; using the source definitions to parse the one or more log files into multiple basic fields; defining one or more extended fields for the one or more log files; and extracting the one or more extended fields from the one or more log files.

[0178] Therefore, some embodiments allow users to add extended field definitions. These are defined schemas seen within fields. A user can perform a create-like action on a source, and then the source and all extensions will become the new user-created source. Extended field definitions define new fields to be created based on the content of fields in a given file. In some embodiments, extended field definitions (and labels) can be applied retroactively. This allows past log data to be processed with later-defined field definitions and labels.

[0179] Figure 14 Example field definitions 1302 are shown. For the first case in the table, the user specifies to view the "Message" file field from the log entry and parsed by the file parser. This Message field will contain text, but the user has identified that they want to capture the SIGNALNAME portion of the message as a new field for this specific message. This new field (SIGNALNAME) is now visible in the captured log entry, visible in the log browser, and can also be stored as part of a metric if a rule is created to do so. In this example, the extended field definition uses the entire contents of the Message. Users can bind either side of their expression using wildcard patterns. For example, the definition could simply be "sent {SIGNALNAME}". The text shown is static text known to never change for this log message. Using [0-9]* in the expression means any number of numeric characters can be present, but they will simply be ignored (because there is no associated field name to name this field). The text following the string "sent" will be assigned to the variable SIGNALNAME.

[0180] The last entry is another example where the user defines two new fields, and in the first field, they also define how to use regular expressions to retrieve this content. Here, there are some characters containing az, AZ, 0-9, or hyphens before the period ".". Everything that matches this expression should be added to a new extended field named HOSTNAME. Anything after the first period will be placed in a new extended field named DOMAINNAME. The HOST field from the file parser still has all the content, but this extended field definition tells us that the feature has added two new fields (HOSTNAME and DOMAINNAME) in addition to the HOST field.

[0181] All extended field definitions using {} delimiters to define new fields use parsed expressions. However, in this example, no parsed expressions are shown except for the HOSTNAME field in the last example. This is because in some implementations, there is a default known regular expression pattern (.)* that means any number of characters. If the user does not provide a regular expression, this expression is used implicitly. If static text exists, the system will take any characters between two static text segments. If there is no static text or characters after the field expression, it is assumed that every character up to the end of the file field is part of the value of the new extended field (such as DOMAINNAME in the last example and CONTENT_LENGTH_LIMIT in the third example). This can cause problems if the log entry has variations that sometimes have additional text. The way to solve this problem is to also define a parsed regular expression for each field, instead of relying on the default implicit regular expression (.)*.

[0182] Some implementations provide the ability to define regular expressions and save them with names. For example, the regular expression used above for hostnames is [a-zA-Z0-9\-]+.

[0183] An example of a saved regular expression could be:

[0184] IP_Address regular expression =>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

[0185] When this saved regular expression is referenced in the extended field definition, the last entry in the table can instead look like this:

[0186] {HOSTNAME:@IP_Address}.{DOMAINNAME}

[0187] The new fields to be created are HOSTNAME and DOMAINNAME. The referenced regular expression created and saved is called IP_Address. When the system performs processing on the agent, it will replace the referenced regular expression "@IP_address" with the following regular expression string:

[0188] "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}"

[0189] By leveraging minor changes to the input string from the user, extended expression definitions can be evaluated directly at the proxy (e.g., using a Perl parsing engine).

[0190] In some embodiments, field reference definitions can be provided. This provides a feature where users can provide lookup tables for SQL queries to transform fields that might have difficult-to-read values ​​into more human-readable content. Three example use cases highlight this need: (a) In log entries, there may be error code fields (core or extended fields) that are simply numbers, in which case users can provide lookup references so that the system adds another new field to store a textual description of the meaning of the error code; (b) In log entries, there may be fields with the target's GUID (core or extended fields), and the system can provide a lookup to the target table using an SQL query, which will create another new field storing the target's display name; (c) As a common use case, IP-to-hostname lookups can also be performed, where the logs may contain IP addresses used for clients, which are then used to look up hostnames.

[0191] As noted above, log types (in some cases in this document, also referred to as "parsers") can be defined to parse log data. One example log type involves a "log parser," which is a parser that can be used to parse the core fields of the source. Another example log type involves a "saved regular expression," which can be used when defining extended field definitions. For example, a hostname can be defined via a regular expression as "[a-zA-Z0-9\-]+". This regular expression can be saved with the name and then used later when the extended field definition is created.

[0192] A log parser is a metadata definition that describes how a log file is read and its contents are extracted into fields. Each log file can be described by a single parser, which breaks down each log entry into its basic fields. Log types can correspond to parsing expression fields, such as Perl regular expressions used for parsing files. When defining a log parser, the author identifies the fields that will always be present in the log file. In this case, the following are the fields present in each entry of the log file above:

[0193] Some fields can be quite complex, meaning that a field may actually contain additional structured content used for some log entries but not others. In some embodiments, this may not be processed by the log file parser because it is inconsistent across each line. Instead, when defining the source, you can define extended fields to break this field down into more fields to handle these cases.

[0194] Profiles can be implemented for various constructs in the system, such as parsers, rules, and sources. Profiles capture differences between data items and different uses and / or versions of user products. For example, source profiles can be created that consider different versions of the monitored user product, such as a source profile that changes the source definition between version 1 and version 2 of the monitored database. Rule profiles can be used to account for differences in the rules to be applied. As another example, parser profiles can be provided to adjust parsing functionality, for example, due to differences in date format between logs from different geographical locations. Different regular expressions can be provided for different parser profiles.

[0195] Regarding log entry delimiters, log files can have content that is known to always be one line per entry (syslog), or they can have content that can span multiple lines (Java Log4j format). The log entry delimiter input allows the user to specify that this log file should always be parsed one line per entry, or to provide a header parsing expression that tells us how to find each new entry. The entry start expression will typically be the same as the first few sections of the parsing expression. The system uses this expression to detect when a new entry is seen instead of the subsequent part of the previous entry.

[0196] For this example, the entry start expression could be:

[0197] ([AZ]{1}[az]{2})\s([0-9]{1,2})\s(0-9}{1,2}): (0-9]{2}): ([0-9]{2})

[0198] This expression searches for a strict month, day, hour, minute, second structure. If the exact sequence of characters is seen, then that "line" is considered the start of a new entry.

[0199] In some embodiments, a table is maintained corresponding to the parsed fields, and this table starts empty (no rows) because the parsed expression is empty. As the user creates a parsed expression, the defined fields are added to this table. This can be achieved by monitoring the text entered in the field and calling a function to determine how many fields have been defined when a ")" is added. The system may ignore some cases of "(" and ")", such as when they are escaped or when they are used with control characters.

[0200] For example, consider the following parsing language:

[0201] ([az]{2})\s([a-z0-9]+)

[0202] In this example, there are two pairs of parentheses "()", which means two fields are defined. The content inside is how to find the field from log entries—the UI that created the parser page for this doesn't care about the content inside the parentheses. This is only evaluated and used on the proxy. The content outside the "(" and ")" is simply static text that helps parse the rows (this UI also doesn't care about this content). To create the correct number of fields in the table, the method counts the number of parentheses "()" pairs in the parsing expression. For each field parsed by the parsing expression, the user provides the field name based on one of the existing public fields.

[0203] Log parser construction

[0204] As noted above, log parsers are typically built manually by individuals who must be intimately familiar with both the exact format of the log files being analyzed and the specific programming infrastructure used to implement the parser. This approach has several drawbacks. For example, this highly manual process requires significant time and resources from skilled technicians, both initially for creating the parser and continuously maintaining it to cope with potential changes in the log file format. Furthermore, this manual approach demands prior knowledge of the log file format, which may not always be available before the log files begin streaming to the log analysis system. Finally, the lack of a suitable parser can potentially halt the log analysis pipeline by analyzing affected log data.

[0205] Some embodiments of the present invention address these problems by providing a method for automatically building log parsers. Instead of requiring manual creation of the log parser content, the log content itself is used to build the parser.

[0206] Figure 15A high-level flowchart illustrating a method for implementing this embodiment of the invention is shown. At 1502, one or more lines of a log file are received for processing. In some embodiments, a log parser is constructed as each line of the log file is received in a streaming manner. This approach allows the parser to be constructed in real-time as the contents of the log file are received. In an alternative embodiment, several lines from the log file may be collected together before processing these lines. This approach would be useful for performing batch processing on log file lines, for example, to implement certain types of processing, such as cluster or group analysis of sets of log file lines that may require a minimum number of collections before processing.

[0207] At position 1504, lines from the log file are analyzed. Analysis is performed to identify specific content and dissimilar sections within the log file lines. As additional lines are processed and more information about the lines is obtained, a higher level of certainty regarding the basic structure of the log file lines can be achieved. It's important to note that the number of lines that need to be examined to generate an accurate parser depends on the complexity and content of the log file lines. However, by using the techniques described in this paper, many log files can be analyzed with only 10-20 lines (or even fewer) to build an acceptablely accurate log parser. This ability to generate a log parser based on examining a relatively small number of lines allows the log parser generation process to be performed in a very time-efficient manner, and thus improves the operation of the computing system itself, as it allows the log parser generation process to be performed in real-time as log file data is streamed to the log analysis system.

[0208] At 1506, a parser is then built based on the analysis of lines from the log file. This is performed, for example, by scanning the contents of one or more sets of logs to construct a regular expression for parsing the logs. This embodiment operates by traversing the selected set of lines to identify commonalities between lines, and then constructing a regular expression that can be used in general to parse log files containing similar lines of log entries.

[0209] Figure 16 A more detailed flowchart of a method for implementing this process, according to some embodiments, is shown. At 1602, a master list is created from the first line of the log being analyzed. This master list includes a mapping structure that maps the contents of log file lines to identified element types within those lines. Examples of these element types include, for example, numeric / integer types, character types, and string types.

[0210] Once the first line has been processed, another line from the log is selected for analysis at 1604. At 1606, analysis is performed by moving through the lines being analyzed to compare against the main list. This action is performed to identify the mutable and immutable parts of the line(s) being analyzed. This can be done by starting from the beginning of the line and moving forward until a mismatch is found. At this point, the process finds the next (or subsequent) common characters. One of the identified common characters is considered a "delimiter," and the middle range is marked as mutable. Note that the size of the middle range can be variable between two lines, so the algorithm should be robust enough to handle this. An example algorithm for identifying the common parts that should be considered delimiters is described in more detail below. The process loops until the end of the line is reached.

[0211] At 1608, the main row can then be updated to reflect both the common and variable parts. Furthermore, the values ​​of the variable parts can be stored if desired.

[0212] At 1610, it is determined whether there are any additional lines to analyze. In some embodiments, it is not necessary to analyze every line in the log to perform log analysis. Instead, only a subset of the lines to be analyzed (e.g., 10 lines) is needed to build the parser. If additional lines need to be analyzed (e.g., only 2 of the 10 lines used for analysis have been processed), then the process returns to 1604–1608 to select and analyze another line from the log.

[0213] If no additional rows need to be analyzed, then at 1612, the updated main row is ready to be processed. As described in more detail below, an example type of processing is assigning at least one variable portion to the data type with the fewest restrictions on the variability of the values ​​detected in that variable portion, for at least one variable portion. Furthermore, commonalities can be identified between rows so that regular expressions can then be constructed from the commonalities. Regular expressions can be generated for immutable portions with placeholders to implement a log parser, where at least two distinct placeholders are associated with different data types.

[0214] Figures 17-1 to 17-21 A diagram of this process is provided. Figure 17-1 An example of log file 1702 is shown. The figure shows two lines within log file 1702. The first line contains the content "N=Bob.", and the second line contains the content "N=Sue.".

[0215] The first action is to select line 1 from log file 1702 to construct master list 1704. For example... Figure 17-2As shown, row 1, corresponding to “N=Bob.”, is selected for processing.

[0216] To construct the main list 1704, each part / unit of the content within line 1 is examined to identify the unit type (also referred to herein as a "parse unit") associated with that part of the line. According to one embodiment, each part of the line is identified based on one of the following parse units: (a) string – this is the default parse unit type corresponding to any type of element that may exist within a string; (b) alpha – this parse unit type corresponds to any number of consecutive alphanumeric elements; (c) integer – this parse unit type corresponds to any number of consecutive integer elements; and / or (d) field rule type – this parse unit type corresponds to a type identified based on a rule definition and can be associated with any number of complex combinations of characters, integers, or symbols. The greater the restriction on the type, the more advantageous it is to choose that type for one or more elements within the line.

[0217] Figures 17-3 to 17-8 The process of constructing the main list 1704 for line 1 of log file 1702 is shown. (As follows) Figure 17-3 As shown, the first character "N" is retrieved and placed in the first position of the main list. A parsing unit type is also identified for this character. In this case, the initial parsing unit type "string" is assigned to this character because the process does not yet have sufficient information to know whether the variability of this element across multiple lines should necessitate assigning it to a different parsing unit type. Therefore, since the main list involves the first line of the log file, this element (and each of the other elements within that line) will be assigned the default "string" parsing unit type, as this parsing unit type covers every possible element type that might exist in the line. Figure 17-4 As shown, the next character "=" from line 1 is also placed in main list 1704 and assigned to the parsing unit type "string". Figure 17-5 As shown, the next character "B" from line 1 is also placed in main list 1704 and assigned to the parsing unit type "string". Figure 17-6 This shows that the next character "o" from line 1 is placed in main list 1704 and assigned to the parsing unit type "string". Figure 17-7 Similarly, the next character "b" from line 1 is placed in main list 1704 and assigned to the parsing unit type "string". Finally, as... Figure 17-8 As shown, the last character "." from line 1 is placed in main list 1704 and assigned to the parsing unit type "string".

[0218] Next, as Figure 17-9As shown, line 2, corresponding to “N=Sue.”, is retrieved from log file 1702 and compared with main list 1704. The content of line 2 can be analyzed element-by-element relative to main list 1704. To illustrate this type of analysis, Figure 17-10 The content of the second row is shown, organized on an element-by-element basis.

[0219] Figures 17-11 to 17-15 This comparative analysis between row 2 and main list 1704 is shown. Figure 17-11 This illustrates the analysis of the position of the first element in line 2 relative to the position of the first element in main list 1704. Here, main list 1704 includes "N" in the first position, which matches the element "N" in the same position in line 2. Therefore, this shows the main list...

[0220] 1704 correctly indicates that the first element of the row has "N" as its content. Similarly, Figure

[0221] 17-12 shows the analysis of the position of the second element, where main list 1704 includes "=" in the second position, which matches the element "=" in the same position within line 2. This indicates that main list 1704 correctly shows the second element of the line with "=" as its content.

[0222] However, as Figure 17-13 As shown, the comparison of the third element position indicates the difference between the contents of main list 1704 and the contents of line 2. Specifically, main list 1704 has a "B" in the third element position, while line 2 includes an "S" in the third element position. This indicates that the third element position is a variable part of line(s).

[0223] The process then continues to identify the next (or subsequent) common element that should be considered the delimiter between the common and variable parts. In the current example, the "." element at the sixth element position is the next common element. The following is combined with... Figure 20 The method that can be used to identify the next common element that should be considered a delimiter is described in more detail. It is important to note that this "skip-forward" method of finding the next common part allows for comparison of any number of characters varying within each of multiple lines, because it does not matter how many characters are skipped in each line to identify the next common character.

[0224] like Figure 17-14As shown, the common parts are the first element position (“N”), the second element position (“=”), and the sixth element position (“.”). The variable parts include the third element position (“B” in the main list and “S” in line 2), the fourth element position (“o” in the main list and “u” in line 2), and the fifth element position (“b” in the main list and “e” in line 2).

[0225] The variable portion forms the analysis scope, within which its contents can be analyzed as a collective group of elements. Furthermore, within the variable portion, common parse unit types can be collapsed together; for example, for the variable portions of both the main list and line 2, this corresponds to "Bob" in the main list and "Sue" from line 2. The most stringent parse unit type associated with these values ​​is the alpha parse unit. Therefore, as... Figure 17-15 As shown, the individual string values ​​used for the variable parts of the main list are replaced with alpha parsing units.

[0226] When considering the content from both the main list and line 2, the parsing unit definition within the main list 1704 can also be used to track the specific content from each line already analyzed. Here, the "Bob" and "Sue" values ​​from lines 1 and 2 for that element position can be included within the parsing unit definition for the alpha parsing unit within the main list. This produces... Figure 17-16 The main list shown is 1704. One reason for tracking these values ​​is to identify content values ​​that can later be used to construct regular expressions. Another reason is to allow reconstruction of any underlying rows from the main list, where the main list essentially provides a compressed collection view of each row used to construct the list.

[0227] Within a single line of a log file, there may be sections that can be interpreted as having meaningful tags / types associated with them. For example, a series of numbers separated by "." values ​​(such as "123.45.67.89") can be identified as an IP address. Therefore, instead of generally representing such sequences as integer, alpha, or string parsing units, a "field rule" type can be constructed, which associates meaningful tags with sequences of these types. Field rule types can include rule definitions related to combinations of characters, integers, and / or symbols associated with a given sequence of interest.

[0228] Figures 17-17 to 17-19 This illustrates the process of identifying the field rule parsing units used for the master list. For example... Figure 17-17As shown, any number of field rules1710 can exist that have been defined for the log analysis system. Each of these field rules corresponds to a different type of sequence for which there is interest in identifying a meaningful type for that sequence. Examples of field rule types include field rules for IP addresses, timestamps, identifiers, etc.

[0229] Figure 17-18 Example field rule 1712 is shown that can be applied to a variable portion of the main list. Field 1712 corresponds to the "Name" type (or name / tag), where, for example, the field rule can be applied to identify a person's name. Field rule 1712 can be associated with regular expression 1714 to identify the portion of the main list that corresponds to the field rule. Here, regular expression 1714 is "[AZ][az]+". This regular expression corresponds to any character sequence where the first character is an uppercase letter (from AZ), followed by any number of subsequent non-uppercase letters (from az). Here, the set of data recorded for the alpha parsing unit of the main list (e.g., "Bob" and "Sue") matches the regular expression of this "Name" field rule. Figure 17-19 As shown, the main list 1704 can then be updated to replace (one or more) alpha parsing unit types with field rule parsing units.

[0230] It's important to note that the master list can be processed in a post-processing action after its construction (after analyzing multiple lines from the log file) to identify field rules. Alternatively, field rules can be identified as lines that are analyzed individually in a streaming manner. In another embodiment, field rules can be identified both during line processing and in subsequent post-processing steps.

[0231] In some embodiments, field rule processing is performed only on the sections of a row that are identified as variable. Alternatively, field rules can be associated with both constant and variable sections of a row.

[0232] Once a sufficient number of lines from the log files have been processed, regular expressions can be constructed from the master list. For example, such as Figure 17-20 As shown, a regular expression "N=([AZ][az]+)\." can be constructed, which is related to the value maintained in the main list 1704. In particular, the constant part of the main list can be directly inserted into the regular expression. The part of the regular expression corresponding to the field rule can be pulled from the regular expression definition of the field rule itself.

[0233] Finally, as Figure 17-21As shown, a log parser 1722 can be constructed for a log file. The log parser 1722 can include regular expressions constructed from a master list, where these regular expressions are associated with line definitions for the log file, which can be used to read and parse each line from the file. Other data / metadata items can be included along with the log parser 1722. For example, the log parser 1722 can be associated with a parser identifier / name, and / or with a file type identifier.

[0234] A key advantage of this approach is its efficiency in detecting distinct subpatterns. The method matches the high-level pattern separately and then uses subpattern detection to attempt to characterize the variable parts that are not fixed components of the high-level pattern. An example subpattern is a key-value pair.

[0235] This approach can also be used to define skeleton parts to construct regular expressions and to build parsers that can assign parts of the expressions to variables. It can also handle patterns below a similarity threshold by assigning variable parts to keep items in the same log consistent (if possible). In some embodiments, parsers are generated for future processing, not just for classification.

[0236] By using this method, logs of any level of complexity can be processed to build log parsers. Consider the following log entries, which are slightly more complex examples:

[0237] [11.22.33.44]Name=Bob Age=27

[0238] [10.20.30.40]Name=Sue Age=30

[0239] This embodiment identifies commonalities between lines by traversing a selected set of lines, and then constructs a regular expression that can be used to generally parse log files containing similar lines of log entries. The master list can be constructed from the first line, which is compared with the second line. The analysis identifies the mutable and immutable parts of the analyzed line(s).

[0240] Suppose the first line (with name = Bob) is used to initially construct the main list. Then the next line (with name = Sue) is analyzed to identify the mutable and immutable parts between the two lines. Here, the first part of the line has the same common character "[". Moving from this character, we can see a range of distinct values ​​in the middle, up to the right square bracket "]". The middle range can optionally be considered as including the common value used for the "." character. The rest of the line can be analyzed similarly so that the "Name=" and "Age=" parts are found to be common, and the range of characters following these common parts is found to be mutable. The main line can be updated to reflect the common and mutable parts. Furthermore, mutable values ​​can also be stored if desired.

[0241] The updated main line can then be processed to construct a regular expression from the common parts. For the example line above, the following identifies the common parts, the variable parts, and the values ​​of the variable parts.

[0242] [{Variable Section 1}]Name = {Variable Section 2}Age = {Variable Section 3}

[0243] Variable segment 1: {11.22.33.44, 10.20.30.40}

[0244] Variable segment 2: {Bob, Sue}

[0245] Variable segment 3: {27, 30}

[0246] This can then be used to construct appropriate regular expressions to parse lines from the logs. For example, the first part of a line containing an IP address could correspond to the following regular expression: "\[[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\.\]". Note that field rules (defined to include this regular expression) can also be used to associate that part of the line with the IP address.

[0247] According to some embodiments, row preprocessing can be performed in advance to prepare log data for processing. For explanation, consider... Figure 19 The example log file shown is 1902. This log file 1902 consists of four lines, where the first line contains "N=", the second line contains "Bob.", the third line contains "N=", and the fourth line contains "Sue.". This is similar to... Figure 17-1 The two lines in file 1702 contain identical content, but in file 1902 they are scattered across four lines. However, the problem lies with the log parser generator, which expects each line to be processed as a single unit, when faced with... Figure 19The log file structure shown in log file 1902 will fail because each log entry actually covers two lines at a time (e.g., the first two lines are the first log entry, and the third and fourth lines are the second log entry).

[0248] Figure 18 The process flow for resolving this type and other types of non-standard line formatting situations is shown. At 1802, one or more lines are identified from the log file for preprocessing.

[0249] At 1804, rows are analyzed for grouping purposes. One possible approach to grouping rows together is to examine their timestamps. For example, in some systems, it's possible that multiple related rows contain only the timestamp used for the first row. In this case, rows are grouped together until another timestamp is identified. Even though each row includes its own timestamp, the commonality of timestamp values ​​allows multiple rows to be identified as parts of a single whole. As an alternative, clustering can be performed to group groups of rows that are assumed to be linked together. Another possibility is to perform pre-classification on the rows to identify the row structure in order to identify rows that should be grouped together.

[0250] Once the groups have been identified, the grouped lines can be considered together for log analysis purposes. According to 1806a, one possible approach is to manipulate the lines so that the grouped content appears within a single line. At 1806b, another approach is to categorize multiple related lines into a single log entry for analytical purposes.

[0251] Figure 19 Section 1904a illustrates a method for manipulating grouped content so that it appears within a single line. Here, the original file 1902 comprises four lines: the first line contains “N=", the second line contains “Bob.”, the third line contains “N=", and the fourth line contains “Sue.”. The intention is to group the first and second lines together, while the third and fourth lines form another group. As shown in section 1904a, the contents of the first and second lines are combined to form a single line with the content “N=Bob.”. Similarly, the contents of the third and fourth lines are combined to form another single line “N=Sue.”. Each of these newly formed single lines is then processed for use in generating the log parser.

[0252] In the alternative method shown in section 1904b, no new rows are created to group multiple related rows together. Instead, for analytical purposes, multiple rows are simply logically grouped together as a single log entry. As shown in section 1904b, the first logical log entry 1906s is formed by the first two rows, and the second logical log entry 1906b is formed by the third and fourth rows. The master list described above is constructed by iterating through the elements of the two rows that involve the common logical log entries. In this case, the "newline" character separating lines within a single log entry can be considered simply another character to be identified and processed within the master list for a given entry.

[0253] As noted earlier, when constructing a log parser, consecutive segments of log lines can be treated as units. Delimiters within a line can be identified to determine the consecutive segments to be identified. Each line can be considered for its common and variable portions, where the common portion can correspond to delimiters that separate one or more elements within the variable portion as a sequence of elements for analysis. For example, in Figure 17-1 In the first line of log file 1702 shown, the variable part “Bob” is delimited on the left by the common part “N=" and on the right by a “.”. Similarly, for line 2, the variable part “Sue” is delimited on the left by the common part “N=" and on the right by a “.”. Therefore, the common element “.” can be identified as a delimiter that separates the preceding variable part (e.g., “Bob” or “Sue”) from the rest of the line for analysis.

[0254] However, if one or more of the common parts should actually be considered as part of the content segment that should be analyzed as a unit, then the simple solution of simply identifying the common parts as delimiters will fail. To illustrate this problem, consider... Figure 21-1 The log file 2102 is shown here. Lines 1 and 2 share a common part at the fourth element position (both lines contain the letter "o" at that position) and at the sixth element position (both lines contain a "." at that position). For a human observer, it's fairly easy to determine that the "." should be the delimiter (not the "o" character), since the letter "o" forms part of the names for both the letter sequences "Bob" and "Tod". However, when performing automated processing of lines, especially when faced with these lines without prior knowledge of their content, identifying the correct delimiter becomes a much more difficult problem. This is especially true because automated processing systems may not have prior knowledge of the "name" type as part of the line structure. In fact, even without prior knowledge of the line structure, the process described in this paper will be used to discover such types within the log line structure.

[0255] According to some embodiments, an inventive method is provided to identify which of one or more common elements within a row should be considered a delimiter. The method operates by traversing the row to identify common elements, wherein a score for each common element is determined by considering a combination of element position and element weight. The element with the highest score (or lowest score, depending on how the score is calculated) can then be identified as the delimiter.

[0256] Figure 20 A flowchart illustrating a method for efficiently identifying correct delimiter elements within a set of log content, according to some embodiments of the present invention, is shown. At point 2002, the content of at least two lines(s) is traversed (e.g., from left to right within the lines) to identify common elements that define the variable portions of the lines.

[0257] At position 2004, the process then iterates through the rest of the line to identify any additional (one or more) common elements within the line. Any number of additional (one or more) common elements may exist.

[0258] In 2008, a score was calculated for each of the identified common elements. First, the position of the common element within the line was determined for the score. The general idea was that, all other things being equal, the first possible delimiter found (e.g., closer to the left when traversing the line from left to right) should be considered first. For example, consider the following line: “Names:Bob Joe Sam”. In this example line, there is a first space between “Bob” and “Joe”, and a second space between “Joe” and “Sam”. In this case, both spaces could be delimiters, but the first space (the left space between “Bob” and “Joe”) should be identified first. Only later, when the delimiter identification process is run again, will the second space (between “Joe” and “Sam”) be identified as the next delimiter. Therefore, when choosing between the first and second spaces, the position of the first space should receive a more prominent score than the second space. This was done in 2010 by providing a scoring factor determined by the element's position within the line. For example, the sum or average of the positions of elements can be identified and associated with the elements.

[0259] Furthermore, the type of the found element should also be taken into account in the delimiter score used for the element. This was done in 2012 by applying a weighting factor to the identified common elements(s) within the row. Figure 22 A graph 2202 is shown, illustrating some example weights that can be applied to identify common elements in some applications of this invention. Specifically, this graph is based on the assumption that lower scores are more likely to identify elements as delimiters. Therefore,Figure 22 Each element type shown is associated with a weighting factor, where element types that are more likely to be delimiters have smaller weighting factors, and element types that are less likely to be delimiters are associated with larger weighting factors. In this example, spaces are most likely to be considered delimiters; therefore, as shown in line 2206, this element type is associated with a very small weighting factor. On the other hand, alphanumeric characters are considered one of the least likely element types to be delimiters; therefore, as shown in line 2210, this element type is associated with a very high weighting factor.

[0260] Weighting factors can also be associated with more complex rules that consider combinations of elements. For example, since a typical IP address has a sequence of numbers punctuated with "." elements, this means that a "." placed between two integer sequences is unlikely to be a delimiter and is more likely to be part of the IP address field. Furthermore, non-integers (e.g., floating-point numbers) can include a decimal point between two numbers (e.g., a "." element between two numeric elements in "2.3"), which also makes the "." element less likely to be a delimiter in this case and more likely to be part of the numeric value. Therefore, as shown in line 2208, such combinations of elements can be associated with a rule that identifies a "." element between two integers, where this rule is associated with a weighting factor to deflect the "." from being a delimiter in this type of element combination.

[0261] As another example rule, consider the case where a given character has already been found to be a delimiter in a line. In this case, the same character later in the same line is very likely to also be a delimiter; for example, an "=" element that was previously found to be a delimiter for key-value pairs in a line is very likely to also be a delimiter for other key-value pairs later in the same line. Therefore, as shown in line 2204, the rule for this case can be associated with a weighting factor to strongly favor elements that were previously identified as delimiters being treated as delimiters again.

[0262] Once the score is calculated, at point 2014, the delimiter can be identified by comparing the scores of the different possible delimiters. For example, if the scoring is configured such that a lower score corresponds to a higher probability as a delimiter, then the element in row(s) with the lowest calculated score will be identified as the delimiter. This process can then be repeated to identify any number of additional delimiters within that row (if they exist).

[0263] It is important to note that while this example calculates the delimiter score where the lowest score is most likely to be a delimiter, the application of the principles disclosed herein can also be used to calculate the score corresponding to the highest score and the most likely delimiter. In this alternative approach, the weighting factors are configured such that element types most likely to be delimiters are associated with higher weighting factors, and element types less likely to be considered delimiters are associated with lower weighting factors.

[0264] Figures 21-1 to 21-5 The process of delimiter recognition is illustrated. Figure 21-1 The log file 2102 is shown, where line 1 corresponds to "N=Bob." and line 2 corresponds to "N=Tod.". For example... Figure 21-2 As shown, the common element "=" on the left is found in both of these rows. The process then iterates through the remaining rows to identify more common elements. Figure 21-3 As shown in the image, "o" is the first public element, and "." is the second public element.

[0265] Then, a delimiter score is calculated for each of the two common elements. Figure 21-4 The process of calculating a score for each of these elements is shown. First, the relative position of the element within each row is calculated. For the element "o", this element exists at position 1 for every row. Therefore, the sum of these position values ​​is 2. For the element ".", this element exists at position 3 for every row. Therefore, the sum of the position values ​​is 6.

[0266] Next, a weighting factor is identified for each element. Figure 22 This shows example weighting factors that can be used in the current scoring calculation. For the element "o", which is an alphanumeric character, it is related to... Figure 22 The weighting factor 100 is associated with the element "." in the chart. Figure 22 The weighting factor 1 in the chart is associated with this.

[0267] In this example, the score is calculated by multiplying the weighting factor by the sum of the locations used for the elements. For the "o" element, the score would therefore be the sum of locations (2) * weighting factor (100) = delimiter score 200. For the "." element, the score is the sum of locations (6) * weighting factor (1) = delimiter score 6.

[0268] The scores are then compared to identify the lowest score, with the element having the lowest score being considered the delimiter. Here, the score of the "." element is lower than that of the "o" element (6 < 200). Therefore, the "." element is identified as the delimiter.

[0269] like Figure 21-5As shown, the parsed segments "Bob" and "Tod" between the "=" element and the delimiter "." can now be identified. In this case, recognizing "." (instead of "o") as the delimiter produces the correct result because it can be seen that although the "o" character is a common element between the two lines, the "o" character is actually part of the parsed segment, not a delimiter.

[0270] Another technique that can be applied in some embodiments is to automatically perform key-value extraction from log data. This approach is particularly useful, for example, for implementing the extended field definitions described above. The current embodiment achieves this by identifying the first and last key-value pair separators in a row, and then using a splitting function to process the content in between to extract the key-value data.

[0271] Figure 23 The flowchart illustrates an example method for performing key-value extraction. The process begins at 2302 by analyzing the line and identifying the first key-value separator it encounters. The key-value separator can be, for example, the "=" character. If no suitable key-value separator is found in the line, the process exits.

[0272] Next, at position 2304, an attempt is made to find a key-value (KV) pair separator (if a KV separator is found, the process stops). This separator can be, for example, a space between different key-value pairs. The process repeats this step to look for additional separators. This thus identifies the range of key-value pairs present in the line for processing.

[0273] The process then returns to the beginning of the identified range to extract the key-value content. At 2306, the key to the left of the first key-value separator is identified. At 2308, the value to the right of the last key-value separator is identified. At 2310, the identified portion of the row is then parsed to identify the key-value pairs. For example, the "split" function from Java or Perl could be used to perform this action. For the current row, this action therefore identifies the key-value pairs for each key-value pair in the row. Therefore, this method can be used to automate key-value extraction. The process iterates through the remaining key-value pairs in the identified range to extract the key-value data for all key-value pairs.

[0274] Figures 24-1 to 24-12 Iterate through this process. Figure 24-1 Example line 2402 from a log file shows the following content: "11 / 12 / 2017 Name=Bob ID=5 Age=21". (The rest of the text appears to be a separate, unrelated sentence.) Figure 24-2 As shown, the process begins by analyzing line 2402 and identifying the first key-value separator seen. This key-value separator can be, for example, the "=" character. Here, searching for the first key-value (KV) separator will find the "=" character within "Name=Bob".

[0275] Next, as Figure 24-3 As shown, the process attempts to find the key-value pair separator. This separator can be, for example, a space between different key-value pairs. In line 2402, this identifies a space between the key-value pair (KVP) "Name=Bob" and the next KVP "ID=5". The process repeats this step to find additional pair separators. For example, as... Figure 24-4 As shown, another pair of separators is found between KVP "ID=5" and the next KVP "Age=21". Figure 24-5 As shown, in line 2402, there are no other key-value pairs to the right of KVP “Age=21” that are identifiers.

[0276] At this point, the scope of analysis for the key-value content extraction process has been identified. For example... Figure 24-6 As shown, the analysis range is from KVP "Name=Bob" to KVP "Age=21".

[0277] The processing will now take place to extract the "key" and "value" for each key-value pair. For example... Figure 24-7 As shown, identify the key to the left of the first key-value separator. In the example row, the first key-value separator is between "Name" and "Bob". The key to the left of this separator is "Name". Next, as... Figure 24-8 As shown, the value to the right of the KV delimiter is identified. In the example row, the value to the right of the delimiter is "Bob". These values ​​can be recorded into any suitable structure such as a database table or a key-value data structure.

[0278] Then proceed to the next key-value pair. For example... Figure 24-9 As shown, identify the key to the left of the next KV delimiter. In the example row, the KV delimiter is between "ID" and "5". Therefore, the key to the left of this delimiter is "ID". Next, as... Figure 24-10 As shown, the value to the right of the key-value delimiter is identified. In the example row, the value to the right of the delimiter is "5". These values ​​are recorded into the appropriate storage structure.

[0279] Then proceed to the last key-value pair. For example... Figure 24-11 As shown, identify the key to the left of the last KV separator. Here, the last KV separator "=" is located between "Age" and "21". The key to the left of this separator is "Age". Next, as... Figure 24-12 As shown, the value to the right of the KV delimiter is identified. Here, the value to the right of the delimiter is "21". These values ​​are recorded into the appropriate storage structure. At this point, all key-value pairs have been identified and extracted into the key-value storage structure.

[0280] Additional optimizations can be performed to efficiently extract key-value content from log files. To explain, consider this again. Figure 24-6 The log file line 2402 shown is an example. The analysis segment identified for this line 2502 is fairly straightforward to process, as it contains only the key-value pairs to be processed. Therefore, iteratively applying the method described above to check the keys and values ​​on either side of the KV delimiter "=" will yield the correct key / value data for each key-value pair.

[0281] However, as Figure 25-1 As shown in line 2504, there is a possibility that non-key-value content may exist within the boundaries of the analysis range. In this case, the date content "11 / 12 / 2017" appears within the analysis range. Therefore, a simple application of the process of only checking the values ​​to the left / right of the KV separator "=" may end with an incorrect assignment of this content to the value content of the previous KV pair (e.g., where the key is "Name" and the value of this key is incorrectly identified as "Bob 11 / 12 / 2017") or with an incorrect assignment of this content to subsequent key content (e.g., the key is incorrectly identified as "11 / 12 / 2017 ID" and the value of this key is identified as "5").

[0282] This problem cannot be corrected simply by considering the whitespace character as a delimiter used to identify keys and values. This is because some keys and / or values ​​may be intended to include spaces as part of the key / value content. To explain this, consider... Figure 26-1 Line 2506 is shown in the image. In this line, the first key-value pair has "Name" as the key. The value of this key is assumed to be "BobSmith", with a space between "Bob" and "Smith". Therefore, any method that always treats spaces as delimiters will incorrectly assign only the "Bob" part as the value, and will not correctly identify "Smith" as part of the value content.

[0283] One possible approach to this problem is to perform preprocessing to categorize the rows so that the key-value portions can be identified for key-value extraction. Using this method, Figure 25-1 The non-KV portion of row 2504 can be correctly identified (e.g., identified as a date field), and the key-value extraction process described above will therefore ignore this portion of the row. Figure 25-2 As shown, this will result in multiple analysis ranges under KV extraction. This method can be similarly used to solve... Figure 26-1In line 2506 shown, the first key-value pair has "Name" as the key, and the value of this key is assumed to be "Bob Smith," with a space between "Bob" and "Smith." Any method that treats spaces as delimiters will incorrectly assign only the "Bob" portion as the value, and will not correctly identify "Smith" as part of the value content. However, preprocessing can be used to recognize the fact that spaces can exist within the value field used for this key-value pair, and as... Figure 26-2 As shown, the correct portion of the line will be identified for each key-value pair, where the space element between "Bob" and "Smith" is considered part of the value field rather than a delimiter. The method described in U.S. Application Serial No. 14 / 863136, filed September 24, 2015, can be used to identify pattern signatures to implement this method, and the entire contents of that U.S. application are incorporated herein by reference. Pattern signatures can correspond to key-value (KV) signatures (e.g., identified KV separators such as "=" separate the identifiable key and value portions), and KV signatures are used to identify the KV analysis portion of a line (and ignore the non-KV portions of the line).

[0284] Another approach is to perform post-processing to correct any problematic assignments of content to key or value fields. This method can be used, for example, to check for incorrect types of values(s) within a key-value field. To illustrate, consider... Figure 25-1 Line 2506 shown, and a simple application of the process of only checking the values ​​to the left / right of the key-value separator "=", might end with an incorrect assignment of the date content to the value content of the previous key-value pair, for example, where the key is "Name" and the value of this key is incorrectly identified as "Bob 11 / 12 / 2017". A rule can be configured for the log analysis system to restrict the scope of type elements to fields that can be identified as the "Name" field, excluding the " / " character from valid names. In this case, post-processing will be able to identify the incorrect portion of the value field based on the " / " character within the "Bob 11 / 12 / 2017" field. The analysis system can then choose to exclude the entire extracted key / value content or choose to correct the erroneous content. Correction can be achieved by scanning forward and / or backward within the line content to identify the correct set of content to be assigned to the key and / or value fields. In this case, the "Bob 11 / 12 / 2017" field will be corrected to associate only "Bob" with the value field of the key "Name".

[0285] Therefore, improved systems, methods, and computer program products for implementing log analysis methods and systems have been described, which can efficiently configure, collect, and analyze log records. In particular, an improved method for automatically generating a log parser by analyzing the line content of logs has been described. Furthermore, an efficient method for extracting key-value content from log content has been described.

[0286] System Architecture Overview

[0287] Figure 27 This is a block diagram of an illustrative computing system 1400 suitable for implementing embodiments of the present invention. The computer system 1400 includes a bus 1406 or other communication mechanism for transmitting information, the bus 1406 or other communication mechanism interconnecting subsystems and devices such as a processor 1407, system memory 1408 (e.g., RAM), static storage device 1409 (e.g., ROM), disk drive 1410 (e.g., magnetic or optical), communication interface 1414 (e.g., modem or Ethernet card), display 1411 (e.g., CRT or LCD), input device 1412 (e.g., keyboard), and cursor controller.

[0288] According to one embodiment of the present invention, computer system 1400 performs a specific operation by executing one or more sequences of one or more instructions contained in system memory 1408 via processor 1407. Such instructions may be read into system memory 1408 from another computer-readable / usable medium, such as static storage device 1409 or disk drive 1410. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the invention. Therefore, embodiments of the invention are not limited to any particular combination of hardware circuitry and / or software. In one embodiment, the term "logic" refers to any combination of software or hardware used to implement all or part of the invention.

[0289] As used herein, the terms "computer-readable medium" or "computer-usable medium" refer to any medium that participates in providing instructions to processor 1407 for execution. Such media can take many forms, including but not limited to non-volatile media and volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as disk drives 1410. Volatile media include dynamic memory, such as system memory 1408.

[0290] Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cassettes, cloud-based storage devices, or any other media from which a computer can read.

[0291] In embodiments of the invention, the execution of the instruction sequence for practicing the invention is performed by a single computer system 1400. According to other embodiments of the invention, two or more computer systems 1400 coupled via a communication link 1415 (e.g., a LAN, PTSN, or wireless network) can coordinate with each other to execute the instruction sequence required for practicing the invention.

[0292] Computer system 1400 can send and receive messages, data, and instructions, including programs, i.e., application code, via communication link 1415 and communication interface 1414. Received program code can be executed by processor 1407 upon receipt and / or stored in disk drive 1410 or other non-volatile storage device for later execution. Data can be accessed from database 1432 maintained in storage device 1431, which is accessed using data interface 1433.

[0293] In the foregoing description, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and alterations can be made to the invention without departing from the broader spirit and scope thereof. For example, the processing flow described above is described with reference to a specific order of processing actions. However, the order of many of the processing actions described may be changed without affecting the scope or operation of the invention. Accordingly, the description and drawings should be considered illustrative rather than restrictive. Furthermore, the illustrated embodiments are not required to have all the aspects or advantages shown. Aspects or advantages described in connection with a particular embodiment are not necessarily limited to that embodiment and may be practiced in any other embodiment, even if not shown so. Moreover, throughout this specification, references to “some embodiments” or “other embodiments” mean that a particular feature, structure, material, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of the phrase “in some embodiments” or “in other embodiments” throughout various places in this specification does not necessarily refer to the same one or more embodiments.

Claims

1. A method for managing log data, comprising: Generate a log collection configuration that indicates the target component type and log source; The log source refers to the location of the log data and includes a file pattern containing fixed and variable parts; In response to determining that a first target component among a plurality of target components in a computing environment is associated with the target component type: the log collection configuration is associated with the first target component; In response to associating the log collection configuration with a first target component: replacing the variable portion of the file pattern associated with the log collection configuration with a first parameter associated with the first target component to generate a first file path indicating a first location of first log data associated with the first target component; Generate the first target-side configuration content, including the first file path; In response to determining that a second target component among the plurality of target components in the computing environment is associated with the target component type: the log collection configuration is associated with the second target component; In response to associating the log collection configuration with a second target component: replacing the variable portion of the file pattern associated with the log collection configuration with a second parameter associated with the second target component to generate a second file path indicating the second location of the second log data associated with the second target component; The first parameter and the second parameter are different; Generate the second target-side configuration content, including the second file path; The configuration content of the first target side is distributed to the first agent associated with the first target component, and the configuration content of the second target side is distributed to the second agent associated with the second target component; Receive first log data captured for the first target component at a first location indicated by the first file path from the first agent; Receive second log data captured for the second target component at a second location indicated by the second file path from the second agent; and The method is performed by at least one device including a hardware processor.

2. The method as described in claim 1, wherein, The log collection configuration further indicates at least one of the following: Basic parser; Log rules applicable to target components of the aforementioned target component type; The identifier of the target component associated with the log collection configuration; and A parser associated with the log source, wherein the parser includes regular expressions to define how to parse log data of a target component of the target component type.

3. The method as described in claim 1, wherein, When the association between the first target component and the log source changes, or when the first target component changes, or when the state of the first agent changes, the log collection configuration is generated, modified, or deleted.

4. The method of claim 1, wherein: The first parameter associated with the first target component includes at least one of the following: first metadata associated with the first target attribute of the first target component, and the first host of the first target component; The second parameter associated with the second target component includes at least one of the following: second metadata associated with the second target attribute of the second target component, and the second host of the second target component.

5. A method for managing log data, comprising: A log collection configuration associated with multiple log collection target components in a storage and computing environment, the log collection configuration including: The first included filename pattern, including (a) a first fixed part and (b) a first variable part, corresponds to a first group of one or more log files to be included in the log set; The first excluded filename pattern includes (a) a second fixed part and (c) a second variable part, corresponding to a second group of one or more log files to be excluded from the log set; At least based on the aforementioned log collection configuration: The first variable portion in the first included filename pattern is replaced with a first value associated with the log collection target component among the plurality of log collection target components to obtain the second included filename pattern; The second variable portion in the first excluded filename pattern is replaced with a second value associated with the log collection target component to obtain a second excluded filename pattern; Specifically, one or more log files in a second group that match a second excluded filename pattern are removed from one or more log files in a first group that match a first group of file locations that match a second included filename pattern, in order to identify one or more log files in a third group to be collected. Information including a second included filename pattern and a second excluded filename pattern for collecting one or more log files in a third group is sent to or stored on the log collection target component.

6. The method of claim 5, wherein, Using a mapping table that maps associations to targets, the first included filename pattern is overridden on a per-target basis.

7. The method of claim 5, wherein, Establish target attributes with metadata to define the first variable part of the first included filename pattern.

8. The method of claim 5, wherein: The log collection configuration is associated with the target component type; At least in response to determining that the log collection target component is associated with the same target component type, a first variable portion of the first included filename pattern is replaced with a first value associated with the log collection target component, and a second variable portion is replaced with a second value associated with the log collection target component.

9. The method of claim 5, wherein: At least in response to detecting the installation of an agent configured to perform log collection on the log collection target component, a first variable portion of a first included filename pattern is replaced with a first value associated with the log collection target component, and a second variable portion of a first excluded filename pattern is replaced with a second value associated with the log collection target component.

10. The method of claim 5, wherein, The second included filename pattern is the complete pathname.

11. A method for managing log data, comprising: Define the log collection rules as follows: A generic filename pattern associated with one or more file locations used to store log information, wherein the generic filename pattern includes a fixed part and a variable part; A group of one or more parsers is configured to parse log information stored at the one or more file locations; Based on the log collection rules: a first target-side configuration is generated, which includes an identifier of a first filename pattern, wherein the first filename pattern is a generic filename pattern in which a variable part is replaced with a first value associated with a first target component among multiple target components in the computing environment; The configuration of the first target side is sent to the first agent installed on the first target component, and the first agent is configured to collect the first log information of the first target component; Based on the log collection rules: a second target-side configuration is generated, which includes an identifier of a second filename pattern, wherein the second filename pattern is a generic filename pattern in which a variable part is replaced with a second value associated with a second target component among the plurality of target components; The first target component and the second target component are different, and the first value and the second value are different; The second target side configuration is sent to the second agent installed on the second target component, and the second agent is configured to collect the second log information of the second target component; Based on the log collection rules: generate a first server-side configuration including the identifiers of the set of one or more parsers; The first server-side configuration is stored on a server configured to at least process the first log information of the first target component and the second log information of the second target component; The method is performed by at least one device including a hardware processor.

12. The method of claim 11, further comprising: The first agent identifies the first filename pattern specified in the configuration on the first target side; The first agent collects first log information of the first target component from the first file location indicated by the first filename pattern; The second agent identifies the second filename pattern specified in the second target-side configuration; The second agent collects the second log information of the second target component from the second file location indicated by the second file name pattern.

13. The method of claim 12, further comprising: The server receives first log information of the first target component from the first agent and second log information of the second target component from the second agent; The server identifies the set of one or more parsers specified in the server-side configuration; The server uses one or more parsers to process the first log information of the first target component and the second log information of the second target component.

14. The method of claim 11, further comprising: Identify multiple server-side configurations stored on the server; The first server-side configuration among the plurality of server-side configurations is determined to correspond to the first target-side configuration for collecting first log information of the first target component; The first log information of the first target component is processed using the first parser specified in the first server-side configuration; The second server-side configuration among the plurality of server-side configurations is determined to correspond to the third target-side configuration used to collect third log information of the third target component among the plurality of target components; The third log information of the third target component is processed using the second parser specified in the second server-side configuration.

15. The method of claim 11, further comprising: in, In response to determining that the log collection rule and the first target component are associated with the same target component type, a first target-side configuration is generated based on the log collection rule.

16. The method of claim 11, further comprising: Synchronization processes are performed periodically to synchronize the log collection rules with the multiple target components.

17. The method of claim 11, wherein, The method of storing the server-side configuration on a server configured to at least process the first log information of the first target component and the second log information of the second target component includes: sending the server-side configuration to the server.

18. The method of claim 11, wherein, The first agent is configured to collect first log information of the first target component from the file location identified by the first filename pattern.

19. A non-transitory machine-readable medium for storing one or more instructions, which, when executed by one or more hardware processors, cause the execution of the method as claimed in any one of claims 1-18.

20. A system for managing log data, comprising: At least one device including a hardware processor, and The device is configured to perform the method as described in any one of claims 1-18.

Images