A log analysis and identification method and system

Abstract

The application discloses a log analysis and identification method and system, relates to the technical field of log analysis, and comprises the following steps: analyzing time-related requirements of deep information analysis targets to determine time requirements of different deep information analysis targets, determining time aggregation granularity, and performing time aggregation on surface information of logs; the time aggregation operation can effectively reduce data redundancy and complexity of the surface information, and provides a reliable basis for subsequent analysis of deep information; surface events associated with the deep information analysis targets are extracted from each aggregated surface information set, deep events are generated, deep information analysis targets are analyzed according to the deep events, the aggregated surface events are associated, time sequences of the associated surface events are taken as the deep events, the deep information analysis targets are analyzed by analyzing the time sequences of the surface events, and the log analysis capacity and adaptability are improved, and subsequent system optimization and monitoring are facilitated.

Description

Technical Field

[0001] This invention relates to the field of log analysis technology, and in particular to a log parsing and identification method and system. Background Technology

[0002] Log data, as a crucial resource recording the system state and critical events during runtime in various software systems, offers a unique perspective for analyzing system problems due to its rich hidden information. To extract deeper information from massive log data, such as system status and abnormal events, log parsing and identification schemes have emerged. These schemes aim to collect, preprocess, extract events, and parse raw log data using automated methods for subsequent, more in-depth analysis. This not only improves the efficiency and accuracy of log analysis but also provides strong support for the health, stability, and availability of the system.

[0003] In existing technologies, log parsing can often only analyze surface information and cannot further analyze the deep information hidden beneath the complex surface information. This results in poor log parsing capabilities and low adaptability, which is not conducive to subsequent system optimization and monitoring.

[0004] Therefore, improving the ability and adaptability of log parsing is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the problems of poor log parsing capabilities and low adaptability in existing technologies that only analyze surface-level information, and to propose a log parsing and identification method, which includes:

[0006] The system acquires the logs to be analyzed and the targets for in-depth information parsing, identifies the surface information and timestamps in the logs to be analyzed, analyzes the time-related requirements of the targets for in-depth information parsing, and determines the time aggregation granularity of each target for in-depth information parsing based on the time-related requirements.

[0007] The logs to be analyzed are aggregated at multiple levels according to the granularity of time aggregation, generating multiple aggregated surface information sets. Each aggregated surface information set corresponds to a deep information parsing target.

[0008] Extract surface events associated with the deep information parsing target from each aggregated surface information set, and generate deep events based on the surface events;

[0009] By analyzing deep events to extract deeper information, we can achieve in-depth logging and analysis, thereby optimizing system operation.

[0010] In some embodiments of this application, identifying surface information and timestamps in the log to be analyzed includes determining the format of the log to be analyzed, extracting the surface information in the log to be analyzed according to the format, and classifying the surface information.

[0011] Extract timestamps from the logs to be analyzed, convert the timestamps to a unified standard, and mark them on the corresponding surface information.

[0012] In some embodiments of this application, the time-related requirements of the deep information parsing target are analyzed, including: splitting each deep information parsing target into multiple sub-targets; determining all behavioral events involved in each sub-target based on surface information; determining the time activity of behavioral events based on the frequency and duration of occurrence of behavioral events; integrating the time activity of all behavioral events involved in each sub-target to determine the time activity of the sub-target; thereby determining the time requirement level of each deep information parsing target; and describing the time-related requirements of the deep information parsing target through the time requirement level.

[0013]

[0014] in, The time requirement level for the i1th deep information parsing target. Let be the transformation coefficient of the i-th deep information analysis target, and n be the number of sub-targets of the i-th deep information analysis target. The combination weights corresponding to the i2th sub-objective are: The temporal activity of the i2th sub-target of the i1th deep information analysis target. for The minimum value in, for The maximum value in, These are the first and second constants of the i1th deep information parsing target, respectively, and [] represents the rounding symbol.

[0015] In some embodiments of this application, the temporal aggregation granularity of each deep information parsing target is determined based on time-related requirements, including:

[0016] An initial time aggregation granularity is determined based on the time requirement level of each deep information analysis target. Surface information is matched with deep information analysis targets to determine the data output and resource allocation of the surface information corresponding to the deep information analysis targets. Relative analysis efficiency is generated based on the data output and resource allocation. The initial time aggregation granularity is adjusted through the relative analysis efficiency to obtain the time aggregation granularity of each deep information analysis target.

[0017] In some embodiments of this application, multi-level time aggregation of surface information in the logs to be analyzed is performed according to the time aggregation granularity, generating multiple aggregated surface information sets, including,

[0018] By matching the target between surface information and deep information, the surface information on the log to be analyzed is aggregated at the corresponding time aggregation granularity. The time aggregation granularity is mapped to the time window size, and the surface information aggregation operation is performed within the time window to obtain multiple aggregated surface information sets at multiple levels.

[0019] In some embodiments of this application, surface events associated with the deep information parsing target are extracted from each aggregated surface information set, and deep events are generated based on the surface events, including:

[0020] Surface events that match the deep information parsing target are denoted as original surface events, and surface events that do not match the deep information parsing target are denoted as other surface events. The event similarity between the original surface events and other surface events is calculated. Based on the event similarity, some other surface events and all original surface events are used as surface events associated with the deep information parsing target.

[0021] The deep information parsing target is associated with the surface events, and the associated surface event sequence is formed according to their respective timestamps. The associated surface event sequence is used as the deep event, and the current state of the deep event is described by a Markov chain.

[0022] In some embodiments of this application, before parsing the deep information parsing target based on deep events, the method further includes,

[0023] For each deep information parsing target, surface events are collected for all possible states, and a sequence of surface events for each possible state of each deep information parsing target is constructed. Markov chains are used to describe the sequence of surface events in the possible states.

[0024] In some embodiments of this application, parsing deep information parsing targets based on deep events includes calculating the similarity and state transition probability between the current state of each deep event and each possible state of the deep information parsing target, and determining the matching degree based on the similarity and state transition probability.

[0025]

[0026] in, Let be the matching degree between the j1-th current state and the j2-th possible state of the deep event. Let be the transition coefficient between the j1-th current state and the j2-th possible state of the deep event. Let be the similarity between the j1-th current state and the j2-th possible state of a deep event. Let be the state transition probability between the j1st current state and the j2nd possible state. It is a constant between the j1st current state and the j2nd possible state;

[0027] The current state of a deep event is determined by the degree of matching between its current state and each possible state of the deep information parsing target, thereby parsing the deep information parsing target.

[0028] Correspondingly, this application also provides a log parsing and identification system, including,

[0029] The first module is used to acquire the logs to be analyzed and the deep information parsing targets, identify the surface information and timestamps in the logs to be analyzed, analyze the time-related requirements involved in the deep information parsing targets, and determine the time aggregation granularity of each deep information parsing target based on the time-related requirements.

[0030] The second module is used to perform multi-level surface information time aggregation on the logs to be analyzed according to the time aggregation granularity, generating multiple aggregated surface information sets, each of which corresponds to a deep information parsing target.

[0031] The third module is used to extract surface events associated with the deep information parsing target from each aggregated surface information set, and generate deep events based on the surface events;

[0032] The fourth module is used to analyze deep information based on deep events, thereby enabling in-depth logging and analysis, and optimizing system operation.

[0033] Compared with the prior art, the beneficial effects of this invention are as follows:

[0034] 1. Analyze the time-related requirements of the deep information parsing targets to determine the time requirements corresponding to different deep information parsing targets, thereby determining the time aggregation granularity. Perform time aggregation on the surface information of the logs. Time aggregation can effectively reduce the data redundancy and complexity of the surface information, providing a reliable foundation for subsequent deep information analysis.

[0035] 2. Extract surface events associated with the deep information parsing target from each aggregated surface information set, generate deep events, and parse the deep information parsing target based on the deep events. Associate the aggregated surface events and use the time series of the associated surface events as deep events. By analyzing the time series of surface events to parse the deep information parsing target, the log parsing capability and adaptability are improved, which helps with subsequent system optimization and monitoring. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a log parsing and identification method proposed in this invention;

[0037] Figure 2 This is a schematic diagram of the structure of a log parsing and identification system proposed in this invention. Detailed Implementation

[0038] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0039] Reference Figure 1 A log parsing and identification method, the scheme includes the following steps:

[0040] Step S101: Obtain the log to be analyzed and the deep information parsing target, identify the surface information and timestamp in the log to be analyzed, analyze the time-related requirements involved in the deep information parsing target, and determine the time aggregation granularity of each deep information parsing target based on the time-related requirements.

[0041] In this embodiment, log data is collected from the system, application, or device. Log data is typically stored in text files and contains key information such as timestamps, event types, and event sources. Deep information analysis targets include user behavior patterns (specific pattern states), security threat detection (specific threat states), and system performance optimization (performance bottleneck states). Different deep information analysis targets may involve different time-related requirements, necessitating targeted analysis and judgment. Here, the time-related requirements of the analysis target refer to the comprehensive time requirements of multiple related contents of the analysis target, and subsequent aggregation operations are performed according to these requirements.

[0042] In some embodiments of this application, identifying surface information and timestamps in the log to be analyzed includes determining the format of the log to be analyzed, extracting the surface information in the log to be analyzed according to the format, and classifying the surface information.

[0043] Extract timestamps from the logs to be analyzed, convert the timestamps to a unified standard, and mark them on the corresponding surface information.

[0044] In this embodiment, the format of the log to be analyzed needs to be defined first. This typically includes the log's text structure, field separators, and the location of key fields. Log formats may vary depending on the system, application, or service, and therefore need to be determined based on the specific circumstances. Based on the defined log format, surface information is extracted from the log using regular expressions, string splitting, or other text processing techniques. Surface information typically includes log level, source IP, user ID, and operation description, which is crucial for initially understanding the log content. The extracted surface information may need to be categorized for subsequent analysis. For example, log levels can be categorized as errors, warnings, information, and debug, and source IPs can be categorized as internal networks and external networks. Categorization simplifies the subsequent analysis process. Timestamps are extracted from the log, which is fundamental for time-related analysis. Timestamps are usually located at the beginning of the log or at a specific position, and their format may vary depending on the system (e.g., ISO8601, UNIX timestamps). To facilitate subsequent time-related analysis, the extracted timestamps need to be converted to a unified standard. For example, all timestamps can be converted to UTC or local time, ensuring consistency in time format. The converted timestamp is then attached to the corresponding surface information, allowing for easy association between the surface information and the timestamp during subsequent analysis. This can be achieved by making the timestamp an attribute of the surface information within the data structure.

[0045] In some embodiments of this application, the time-related requirements of the deep information parsing target are analyzed, including: splitting each deep information parsing target into multiple sub-targets; determining all behavioral events involved in each sub-target based on surface information; determining the time activity of behavioral events based on the frequency and duration of occurrence of behavioral events; integrating the time activity of all behavioral events involved in each sub-target to determine the time activity of the sub-target; thereby determining the time requirement level of each deep information parsing target; and describing the time-related requirements of the deep information parsing target through the time requirement level.

[0046]

[0047] in, The time requirement level for the j1th deep information analysis target is... Let be the transformation coefficient of the i-th deep information analysis target, and n be the number of sub-targets of the i-th deep information analysis target. The combination weights corresponding to the i2th sub-objective are: The temporal activity of the i2th sub-target of the i1th deep information analysis target. for The minimum value in, for The maximum value in, These are the first and second constants of the i1th deep information parsing target, respectively, and [] represents the rounding symbol.

[0048] In this embodiment, each deep information analysis target is broken down into multiple sub-targets. For example, analyzing user behavior patterns can be broken down into multiple sub-targets, including login behavior, browsing behavior, and purchasing behavior. Each behavior may involve multiple behavioral events. The temporal activity of a behavioral event is determined based on its frequency and duration. The temporal activity of a behavioral event is determined by combining its frequency and duration. The temporal activity of all behavioral events involved in each sub-target is then integrated to determine the temporal activity of that sub-target. This integration can be done using methods such as weighted summation or weighted average.

[0049] In this embodiment, Indicates according to The average of the maximum and minimum values ​​is used to correct the sum of the temporal activity of the sub-targets. This is to balance the size of the correction function. This is to balance the different levels of time requirements.

[0050] It should be noted that the behavioral events here refer to the events in the logs before aggregation, while the subsequent surface events refer to the statistical and comprehensive surface events extracted after aggregation.

[0051] In some embodiments of this application, the temporal aggregation granularity of each deep information parsing target is determined based on time-related requirements, including:

[0052] An initial time aggregation granularity is determined based on the time requirement level of each deep information analysis target. Surface information is matched with deep information analysis targets to determine the data output and resource allocation of the surface information corresponding to the deep information analysis targets. Relative analysis efficiency is generated based on the data output and resource allocation. The initial time aggregation granularity is adjusted through the relative analysis efficiency to obtain the time aggregation granularity of each deep information analysis target.

[0053] In this embodiment, an initial time aggregation granularity is determined based on the time requirement level of each deep information parsing target, obtained through a preset mapping relationship. Surface information related to each deep information parsing target is identified. This typically requires a deep understanding of log formats and content. A matching relationship is established between surface information and deep information parsing targets to ensure that each deep information parsing target can extract the required data from the corresponding surface information. Based on the matching relationship, the data output of the surface information corresponding to each deep information parsing target is calculated. Data output can be expressed as the number of log entries or the amount of data generated within a certain time window. The resource requirements for processing each deep information parsing target are evaluated, including computing resources, storage resources, and network bandwidth. The resource allocation should match the data output and the complexity of the parsing target. An analysis efficiency index (data volume, resource allocation) is defined to measure the analysis efficiency of each deep information parsing target. Analysis efficiency can be expressed as the data output or parsing speed processed under given resources. The analysis efficiency of each deep information parsing target is compared with a benchmark or average value to calculate the relative analysis efficiency. Relative analysis efficiency reflects the performance of each analytical target during the analysis process. The initial time aggregation granularity is adjusted using relative analysis efficiency; different relative analysis efficiencies correspond to different adjustment coefficients, which are then multiplied by the initial time aggregation granularity for adjustment.

[0054] Step S102: Perform multi-level surface information time aggregation on the log to be analyzed according to the time aggregation granularity to generate multiple aggregated surface information sets, each of which corresponds to a deep information parsing target.

[0055] In some embodiments of this application, multi-level time aggregation of surface information in the logs to be analyzed is performed according to the time aggregation granularity, generating multiple aggregated surface information sets, including,

[0056] By matching the target between surface information and deep information, the surface information on the log to be analyzed is aggregated at the corresponding time aggregation granularity. The time aggregation granularity is mapped to the time window size, and the surface information aggregation operation is performed within the time window to obtain multiple aggregated surface information sets at multiple levels.

[0057] In this embodiment, different levels of time aggregation granularity are determined based on the analysis requirements, such as second-level, minute-level, hour-level, and day-level. Each time aggregation granularity is mapped to a corresponding time window size. For example, a few-second aggregation may correspond to a few-second time window, a few-minute aggregation may correspond to several 60-second time windows, and so on. Within the time window, aggregation operations are performed on the matched surface information. Aggregation operations may include counting (e.g., the number of requests from a specific IP address within the time window), summing (e.g., the total number of accesses by a specific user within the time window), and averaging (e.g., the average response time of a specific request type within the time window). Each deep information parsing target corresponds to an aggregated surface information set.

[0058] Step S103: Extract surface events associated with the deep information parsing target from each aggregated surface information set, and generate deep events based on the surface events.

[0059] In this embodiment, for surface events associated with the deep information parsing target, in addition to surface events under the matching relationship, other surface events are filtered by event similarity before the matching relationship is completed.

[0060] In some embodiments of this application, surface events associated with the deep information parsing target are extracted from each aggregated surface information set, and deep events are generated based on the surface events. This includes recording surface events that have a matching relationship with the deep information parsing target as original surface events, recording surface events that do not have a matching relationship with the deep information parsing target as other surface events, calculating the event similarity between the original surface events and other surface events, and using the event similarity to identify some other surface events and all original surface events as surface events associated with the deep information parsing target.

[0061] The deep information parsing target is associated with the surface events, and the associated surface event sequence is formed according to their respective timestamps. The associated surface event sequence is used as the deep event, and the current state of the deep event is described by a Markov chain.

[0062] In this embodiment, event similarity can be cosine similarity, etc., and an association standard is determined based on the calculation results of event similarity and analysis requirements. For example, a similarity threshold can be set; when the similarity between other surface events and the original surface event exceeds this threshold, they are considered to be associated with the deep information parsing target. The surface events associated with the deep information parsing target are sorted according to their respective timestamps to form an associated surface event sequence. This associated surface event sequence is considered a deep event. A deep event is a higher-level event representation that contains multiple surface events associated with the deep information parsing target and their temporal order. In this embodiment, a Markov chain is chosen to describe the current state of the deep event. A Markov chain is a stochastic process that assumes the current state depends only on the previous state and is independent of earlier states. By constructing a Markov chain model, the state transition process of deep events can be simulated and analyzed.

[0063] In some embodiments of this application, before parsing the deep information parsing target based on deep events, the method further includes,

[0064] For each deep information parsing target, surface events are collected for all possible states, and a sequence of surface events for each possible state of each deep information parsing target is constructed. Markov chains are used to describe the sequence of surface events in the possible states.

[0065] In this embodiment, all possible states of the target being analyzed refer to all possible states under that target. For example, all possible states of a user behavior pattern include normal, abnormal, and suspicious states. For each possible state of the deep information analysis target, surface events related to that state are collected. These events can be obtained from various sources such as log data, sensor data, and user feedback. The collected surface events are organized, including removing duplicate events, correcting erroneous events, and filling in missing events, to ensure the accuracy and completeness of the event data. The surface events in each possible state are sorted according to timestamps or other suitable sorting criteria to form an event sequence. A constructed Markov chain model is used to describe the surface event sequence in each possible state. By simulating the state transition process, future possible surface event sequences can be predicted, or the rationality of already occurred surface event sequences can be analyzed.

[0066] Step S104: Analyze the deep information parsing target based on deep events, thereby realizing in-depth mining and analysis of logs and optimizing system operation.

[0067] In this embodiment, deep events are time series of surface events, which is a relatively general and abstract concept. Analyzing this sequence determines the specific state of the target being analyzed. Here, analysis refers to calculating the similarity and state transition probability of the surface event sequence, thereby determining the specific state of the current target being analyzed as reflected in the log.

[0068] In some embodiments of this application, parsing deep information parsing targets based on deep events includes calculating the similarity and state transition probability between the current state of each deep event and each possible state of the deep information parsing target, and determining the matching degree based on the similarity and state transition probability.

[0069]

[0070] in, Let be the matching degree between the j1-th current state and the j2-th possible state of the deep event. Let be the transition coefficient between the j1-th current state and the j2-th possible state of the deep event. Let be the similarity between the j1-th current state and the j2-th possible state of a deep event. Let be the state transition probability between the j1st current state and the j2nd possible state. It is a constant between the j1st current state and the j2nd possible state;

[0071] The current state of a deep event is determined by the degree of matching between its current state and each possible state of the deep information parsing target, thereby parsing the deep information parsing target.

[0072] In this embodiment, the similarity between the current state of each deep event and each possible state of the deep information parsing target is calculated. This similarity includes two types: similarity between surface events and similarity between the order of surface events. The overall similarity is determined by combining these two types of similarity. The state transition probability describes the probability of transitioning from the current state to each possible state. This represents the correction for similarity based on state transition probabilities; the higher the state transition probability, the higher the similarity correction. This is to balance the size of the correction function for the state transition probability.

[0073] Correspondingly, this application also provides a log parsing and identification system, such as... Figure 2 As shown, including,

[0074] The first module is used to acquire the logs to be analyzed and the deep information parsing targets, identify the surface information and timestamps in the logs to be analyzed, analyze the time-related requirements involved in the deep information parsing targets, and determine the time aggregation granularity of each deep information parsing target based on the time-related requirements.

[0075] The second module is used to perform multi-level surface information time aggregation on the logs to be analyzed according to the time aggregation granularity, generating multiple aggregated surface information sets, each of which corresponds to a deep information parsing target.

[0076] The third module is used to extract surface events associated with the deep information parsing target from each aggregated surface information set, and generate deep events based on the surface events;

[0077] The fourth module is used to analyze deep information based on deep events, thereby enabling in-depth logging and analysis, and optimizing system operation.

[0078] Compared with the prior art, the beneficial effects of this invention are as follows:

[0079] 1. Analyze the time-related requirements of the deep information parsing targets to determine the time requirements corresponding to different deep information parsing targets, thereby determining the time aggregation granularity. Perform time aggregation on the surface information of the logs. Time aggregation can effectively reduce the data redundancy and complexity of the surface information, providing a reliable foundation for subsequent deep information analysis.

[0080] 2. Extract surface events associated with the deep information parsing target from each aggregated surface information set, generate deep events, and parse the deep information parsing target based on the deep events. Associate the aggregated surface events and use the time series of the associated surface events as deep events. By analyzing the time series of surface events to parse the deep information parsing target, the log parsing capability and adaptability are improved, which helps with subsequent system optimization and monitoring.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0082] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0083] Those skilled in the art will understand that the modules in the system of the implementation scenario can be distributed throughout the system of the implementation scenario as described, or they can be modified to reside in one or more systems different from this implementation scenario. The modules of the above-mentioned implementation scenario can be merged into one module, or they can be further divided into multiple sub-modules.

[0084] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A log parsing and identification method, characterized in that, include, Obtain the log to be analyzed and the target of deep information parsing, identify the surface information and timestamps in the log to be analyzed, analyze the time-related requirements involved in the target of deep information parsing, and determine the time aggregation granularity of each target of deep information parsing based on the time-related requirements; The logs to be analyzed are aggregated at multiple levels according to the granularity of time aggregation, generating multiple aggregated surface information sets. Each aggregated surface information set corresponds to a deep information parsing target. Extract surface events associated with the deep information parsing target from each aggregated surface information set, and generate deep events based on the surface events; By analyzing deep events to extract deeper information, we can achieve in-depth logging and analysis, thereby optimizing system operation. in, And analyze the time-related requirements involved in the deep information parsing target, including, Each deep information analysis target is broken down into multiple sub-targets. Based on surface information, all behavioral events involved in each sub-target are determined. The temporal activity of behavioral events is determined based on their frequency and duration. The temporal activity of all behavioral events involved in each sub-target is integrated to determine the temporal activity of that sub-target. This determines the time requirement level of each deep information analysis target. The time requirement level is used to describe the time-related requirements involved in the deep information analysis target. ； in, For the first The time requirement level for deep information analysis targets For the first The conversion coefficient of a deep information analysis target. For the first The number of sub-targets of a deep information analysis target. For the first The combination weights corresponding to each sub-objective For the first The first deep information analysis target The time activity level of each sub-target for The minimum value in, for The maximum value in, , The first The first and second constants of the deep information analysis target, [] denote the rounding symbol.

2. The log parsing and identification method according to claim 1, characterized in that, Identify the surface information and timestamps in the log to be analyzed, including: Determine the format of the log to be analyzed, extract the surface information from the log according to the format, and classify the surface information. Extract timestamps from the logs to be analyzed, convert the timestamps to a unified standard, and mark them on the corresponding surface information.

3. The log parsing and identification method according to claim 1, characterized in that, And based on time-related requirements, the time aggregation granularity of each deep information parsing target is determined, including, An initial time aggregation granularity is determined based on the time requirement level of each deep information analysis target. Surface information is matched with deep information analysis targets to determine the data output and resource allocation of the surface information corresponding to the deep information analysis targets. Relative analysis efficiency is generated based on the data output and resource allocation. The initial time aggregation granularity is adjusted through the relative analysis efficiency to obtain the time aggregation granularity of each deep information analysis target.

4. The log parsing and identification method according to claim 3, characterized in that, The logs to be analyzed are subjected to multi-level time aggregation of surface information according to the granularity of time aggregation, generating multiple aggregated surface information sets, including, By matching the target between surface information and deep information, the surface information on the log to be analyzed is aggregated at the corresponding time aggregation granularity. The time aggregation granularity is mapped to the time window size, and the surface information aggregation operation is performed within the time window to obtain multiple aggregated surface information sets at multiple levels.

5. The log parsing and identification method according to claim 3, characterized in that, In each aggregated surface information set, surface events associated with the deep information parsing target are extracted. Based on these surface events, deep events are generated, including... Surface events that match the deep information parsing target are denoted as original surface events, and surface events that do not match the deep information parsing target are denoted as other surface events. The event similarity between the original surface events and other surface events is calculated. Based on the event similarity, some other surface events and all original surface events are used as surface events associated with the deep information parsing target. The deep information parsing target is associated with the timestamps of the surface events to form a sequence of related surface events. The sequence of related surface events is used as deep events, and the current state of the deep events is described by a Markov chain.

6. The log parsing and identification method according to claim 5, characterized in that, Before parsing the deep information parsing target based on deep events, the method further includes, For each deep information parsing target, surface events are collected for all possible states, and a sequence of surface events for each possible state of each deep information parsing target is constructed. Markov chains are used to describe the sequence of surface events in the possible states.

7. The log parsing and identification method according to claim 6, characterized in that, Analyze deep information based on deep events to understand the target. include, Calculate the similarity and state transition probability between the current state of each deep event and each possible state of the deep information parsing target, and determine the matching degree based on the similarity and state transition probability; ； in, For the first deep event The current state and the first The degree of matching between the possible states For the first deep event The current state and the first Transition coefficients between possible states For the first deep event The current state and the first The similarity between possible states For the first The current state and the first The state transition probability between 10 possible states For the first The current state and the first A constant between the possible states; The current state of a deep event is determined by the degree of matching between its current state and each possible state of the deep information parsing target, thereby parsing the deep information parsing target.

8. A log parsing and identification system, characterized in that, The system is used to implement the log parsing and identification method as described in any one of claims 1-7, and includes, The first module is used to acquire the logs to be analyzed and the deep information parsing targets, identify the surface information and timestamps in the logs to be analyzed, analyze the time-related requirements involved in the deep information parsing targets, and determine the time aggregation granularity of each deep information parsing target based on the time-related requirements. The second module is used to perform multi-level surface information time aggregation on the logs to be analyzed according to the time aggregation granularity, generating multiple aggregated surface information sets, each of which corresponds to a deep information parsing target. The third module is used to extract surface events associated with the deep information parsing target from each aggregated surface information set, and generate deep events based on the surface events; The fourth module is used to analyze deep information based on deep events, thereby enabling in-depth logging and analysis, and optimizing system operation.

Images