Log analysis and identification method and system

Abstract

The invention discloses a log analysis identification method and system, and relates to the technical field of log analysis, and the method comprises the steps: analyzing time-related demands related to deep information analysis targets to determine time demand conditions corresponding to different deep information analysis targets so as to determine time aggregation granularity, carrying out time aggregation on surface information of a log, and carrying out time aggregation on the surface information of the log; through time aggregation operation, data redundancy and complexity of surface information can be effectively reduced, and a reliable basis is provided for subsequent deep information analysis. Extracting a surface layer event associated with a deep information analysis target from each aggregated surface layer information set, generating a deep layer event, analyzing the deep information analysis target according to the deep layer event, associating the aggregated surface layer events, taking a time sequence of the associated surface layer events as the deep layer event, and analyzing the deep layer information analysis target according to the deep layer event. The deep information analysis target is analyzed by analyzing the time sequence of the surface event, so that the log analysis capability and adaptability are improved, and subsequent system optimization and monitoring are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of log analysis, and in particular, to a log parsing and recognition method and system. Background Art

[0002] Log data, as an important resource for recording the system status and key events during the operation of various software systems, the rich information hidden in it provides a unique perspective for analyzing system problems. In order to mine deep information from massive log data, such as system status, abnormal events, etc., log parsing and recognition solutions have emerged. This solution aims to collect, preprocess, event extraction, and parse the original log data through automated means for subsequent in-depth analysis. This not only improves the efficiency and accuracy of log analysis but also provides strong support for the health, stability, and availability of the system.

[0003] In the prior art, the parsing of logs often can only analyze surface information and cannot further analyze the deep information hidden under complex surface information, resulting in poor log parsing ability and low adaptability, which is not conducive to the subsequent optimization and monitoring of the system.

[0004] Therefore, how to improve the ability and adaptability of log parsing is a technical problem to be solved currently. Summary of the Invention

[0005] The object of the present invention is to solve the problem of poor log parsing ability and low adaptability caused by only analyzing surface information in the prior art, and a log parsing and recognition method is proposed, which includes, Obtain the log to be analyzed and the deep information parsing target, identify the surface information and timestamp in the log to be analyzed, analyze the time-related requirements involved in the deep information parsing target, and determine the time aggregation granularity of each deep information parsing target according to the time-related requirements.

[0006] Perform multi-level surface information time aggregation on the log to be analyzed according to the time aggregation granularity, generate multiple aggregated surface information sets, and each aggregated surface information set corresponds to a deep information parsing target; Extract the surface events associated with the deep information parsing target in each aggregated surface information set, and generate deep events according to the surface events; Parse the deep information parsing target according to the deep events, so as to achieve in-depth mining and analysis of the log and optimize the system operation.

[0007] In some embodiments of the present application, identifying the surface information and timestamp in the log to be analyzed includes determining the format of the log to be analyzed, extracting the surface information in the log to be analyzed according to the format, and classifying the surface information; Extract timestamps from the logs to be analyzed, perform conversions with a unified standard on the timestamps, and mark them on the corresponding surface information.

[0008] In some embodiments of the present application, analyze the time-related requirements involved in the deep information parsing target, including splitting each deep information parsing target into multiple sub-targets, determining all the behavioral events involved in each sub-target according to the surface information, determining the time activity of the behavioral events according to the occurrence frequency and duration of the behavioral events, integrating the time activities of all the behavioral events involved in each sub-target to determine the time activity of the sub-target, and thus determining the time requirement level of each deep information parsing target, and describing the time-related requirements involved in the deep information parsing target through the time requirement level; Among them, is the time requirement level of the i1-th deep information parsing target, is the conversion coefficient of the i1-th deep information parsing target, n is the number of sub-targets of the i1-th deep information parsing target, is the combination weight corresponding to the i2-th sub-target, is the time activity of the i2-th sub-target of the i1-th deep information parsing target, is the minimum value in, is the maximum value in, are respectively the first constant and the second constant of the i1-th deep information parsing target, and [] represents the rounding symbol.

[0009] In some embodiments of the present application, determine the time aggregation granularity of each deep information parsing target according to the time-related requirements, including, Determine an initial time aggregation granularity according to the time requirement level of each deep information parsing target, match the surface information with the deep information parsing target, determine the data output and resource allocation amount of the surface information corresponding to the deep information parsing target, generate a relative analysis efficiency based on the data output and resource allocation amount, and adjust the initial time aggregation granularity through the relative analysis efficiency to obtain the time aggregation granularity of each deep information parsing target.

[0010] In some embodiments of the present application, perform multi-level surface information time aggregation on the logs to be analyzed according to the time aggregation granularity, and generate multiple aggregated surface information sets, including, Perform surface information time aggregation with a corresponding time aggregation granularity on the surface information in the log to be analyzed by matching the relationship between the surface information and the deep information parsing target. Map the time aggregation granularity to the time window size, and perform the aggregation operation of the surface information within the time window to obtain multiple aggregated surface information sets at multiple levels.

[0011] In some embodiments of the present application, extract surface events associated with the deep information parsing target from each aggregated surface information set, and generate deep events according to the surface events, including: Denote the surface events that have a matching relationship with the deep information parsing target as original surface events, and denote the surface events that do not have a matching relationship with the deep information parsing target as other surface events. Calculate the event similarity between the original surface events and the other surface events, and use the event similarity to take some of the other surface events and all of the original surface events as the surface events associated with the deep information parsing target; Form an associated surface event sequence according to the timestamps of the surface events associated with the deep information parsing target, use the associated surface event sequence as the deep event, and describe the current state of the deep event through a Markov chain.

[0012] In some embodiments of the present application, before parsing the deep information parsing target according to the deep event, the method further includes: Collect surface events for all possible states of each deep information parsing target, construct a surface event sequence for each possible state of each deep information parsing target, and describe the surface event sequence in the possible state through a Markov chain.

[0013] In some embodiments of the present application, parsing the deep information parsing target according to the deep event includes calculating the similarity and state transition probability between the current state of each deep event and each possible state of the deep information parsing target, and determining the matching degree based on the similarity and state transition probability; Wherein, is the matching degree between the j1-th current state and the j2-th possible state of the deep event, is the conversion coefficient between the j1-th current state and the j2-th possible state of the deep event, is the similarity between the j1-th current state and the j2-th possible state of the deep event, is the state transition probability between the j1-th current state and the j2-th possible state, is the constant between the j1-th current state and the j2-th possible state; The current state of the deep event is determined according to the matching degree between the current state of the deep event and each possible state of the deep information parsing target, so as to parse the deep information parsing target.

[0014] Correspondingly, the present application further provides a log parsing and recognition system, including A first module, configured to obtain the log to be analyzed and the deep information parsing target, identify the surface information and timestamp in the log to be analyzed, analyze the time-related requirements involved in the deep information parsing target, and determine the time aggregation granularity of each deep information parsing target according to the time-related requirements; A second module, configured to perform multi-level surface information time aggregation on the log to be analyzed according to the time aggregation granularity, generate multiple aggregated surface information sets, and each aggregated surface information set corresponds to a deep information parsing target; A third module, configured to extract the surface events associated with the deep information parsing target in each aggregated surface information set, and generate deep events according to the surface events; A fourth module, configured to parse the deep information parsing target according to the deep events, so as to realize the in-depth mining and analysis of the log and optimize the system operation.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. Analyze the time-related requirements involved in the deep information parsing target to determine the time requirement situations corresponding to different deep information parsing targets, so as to determine the time aggregation granularity, perform time aggregation on the surface information of the log, and the time aggregation operation can effectively reduce the data redundancy and complexity of the surface information, providing a reliable basis for the subsequent analysis of the deep information.

[0016] 2. Extract the surface events associated with the deep information parsing target in each aggregated surface information set, generate deep events, parse the deep information parsing target according to the deep events, associate the aggregated surface events, use the time series of the associated surface events as the deep events, and parse the deep information parsing target by analyzing the time series of the surface events, improving the log parsing ability and adaptability, and helping the subsequent system optimization and monitoring. Brief Description of the Drawings

[0017] Figure 1 It is a schematic flowchart of a log parsing and recognition method proposed by the present invention; Figure 2 It is a schematic structural diagram of a log parsing and recognition system proposed by the present invention. Detailed Embodiments

[0018] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0019] Referring to Figure 1 , a log parsing and recognition method, the solution includes the following steps, Step S101, obtain the log to be analyzed and the deep information parsing target, identify the surface information and timestamp in the log to be analyzed, analyze the time-related requirements involved in the deep information parsing target, and determine the time aggregation granularity of each deep information parsing target according to the time-related requirements.

[0020] In this embodiment, log data is collected from a system, an application, or a device. The log data is usually stored in the form of a text file and contains key information such as timestamps, event types, and event sources. The deep information parsing targets include user behavior patterns (specific pattern states), detecting security threats (specific threat states), optimizing system performance (performance bottleneck states), etc. The time-related requirements involved in different deep information parsing targets may be different, and targeted analysis and judgment are required. The time-related requirements of the parsing target here refer to the comprehensive requirements of multiple related contents of the parsing target for time, and subsequent aggregation operations are performed according to this requirement.

[0021] In some embodiments of the present application, identifying the surface information and timestamp in the log to be analyzed includes determining the format of the log to be analyzed, extracting the surface information in the log to be analyzed according to the format, and classifying the surface information; Extract the timestamp from the log to be analyzed, perform a unified standard conversion on the timestamp, and mark it on the corresponding surface information.

[0022] In this embodiment, first, it is necessary to clarify the format of the log to be analyzed. This usually includes the text structure of the log, field delimiters, positions of key fields, etc. The log format may vary depending on the system, application, or service, so it needs to be determined according to the actual situation. According to the determined log format, use regular expressions, string splitting, or other text processing techniques to extract the surface information from the log. The surface information usually includes log level, source IP, user ID, operation description, etc. These information are very important for initially understanding the log content. The extracted surface information may need to be classified for subsequent analysis. For example, the log level can be classified as error, warning, information, debug, etc., and the source IP can be classified as internal network, external network, etc. Classification processing helps to simplify the subsequent analysis process. Extract the timestamp from the log, which is the basis for time-related analysis. The timestamp is usually located at the beginning of the log or a specific position, and the format may vary depending on the system (such as ISO8601, UNIX timestamp, etc.). For the convenience of subsequent time-related analysis, the extracted timestamp needs to be converted to a unified standard. For example, all timestamps can be converted to UTC time or local time to ensure the consistency of the time format. Mark the converted timestamp on the corresponding surface information so that the surface information and the timestamp can be easily associated during the subsequent analysis process. This can be achieved by taking the timestamp as an attribute of the surface information in the data structure.

[0023] In some embodiments of the present application, analyze the time-related requirements involved in the deep information parsing target, including splitting each deep information parsing target into multiple sub-targets, determining all behavior events involved in each sub-target according to the surface information, determining the time activity of the behavior events according to the occurrence frequency and duration of the behavior events, integrating the time activity of all behavior events involved in each sub-target to determine the time activity of the sub-target, and thus determining the time requirement level of each deep information parsing target, and describing the time-related requirements involved in the deep information parsing target through the time requirement level; Among them, is the time requirement level of the j1-th deep information parsing target, is the conversion coefficient of the i1-th deep information parsing target, n is the number of sub-targets of the i1-th deep information parsing target, is the combination weight corresponding to the i2-th sub-target, is the time activity of the i2-th sub-target of the i1-th deep information parsing target, is the minimum value in, is the maximum value in, They are the first constant and the second constant of the i1-th deep information parsing target respectively, and [] represents the rounding symbol.

[0024] In this embodiment, each deep information parsing target is split into multiple sub-targets. For example, analyzing the user behavior pattern can be split into multiple sub-targets, including login behavior, browsing behavior, purchase behavior, etc. Each behavior may involve multiple behavior events. The time activity of the behavior event is determined according to the occurrence frequency and duration of the behavior event. The time activity of the behavior event is determined by comprehensively considering the occurrence frequency and duration. The time activities of all behavior events involved in each sub-target are integrated to determine the time activity of the sub-target. This integration method can be weighted summation or weighted average, etc.

[0025] In this embodiment, It means to correct the sum of the time activities of the sub-targets according to the average of the maximum and minimum values of to balance the magnitude of the correction function, is to balance the magnitude of the time requirement level.

[0026] It should be noted that the behavior events here are the event situations of the pre-aggregation logs, and the subsequent surface events refer to the statistical and comprehensive surface events extracted after aggregation.

[0027] In some embodiments of the present application, the time aggregation granularity of each deep information parsing target is determined according to time-related requirements, including An initial time aggregation granularity is determined according to the time requirement level of each deep information parsing target. The surface information is matched with the deep information parsing target to determine the data output and resource allocation amount of the surface information corresponding to the deep information parsing target. A relative analysis efficiency is generated based on the data output and resource allocation amount, and the initial time aggregation granularity is adjusted through the relative analysis efficiency to obtain the time aggregation granularity of each deep information parsing target.

[0028] In this embodiment, an initial time aggregation granularity is determined according to the time requirement level of each deep information parsing target, and the initial time aggregation granularity is obtained through a preset mapping relationship. Identify the surface information related to each deep information parsing target. This usually requires an in-depth understanding of the log format and content. Establish a matching relationship between the surface information and the deep information parsing target to ensure that each deep information parsing target can extract the required data from the corresponding surface information. According to the matching relationship, calculate the data output of the surface information corresponding to each deep information parsing target. The data output can be expressed as the number of log entries or the amount of data generated within a certain time window. Evaluate the amount of resources required to process each deep information parsing target, including computing resources, storage resources, and network bandwidth, etc. The resource allocation amount should match the data output and the complexity of the parsing target. Define an analysis efficiency metric (data volume, resource allocation amount) to measure the analysis efficiency of each deep information parsing target. The analysis efficiency can be expressed as the data output or parsing speed processed under given resources. Compare the analysis efficiency of each deep information parsing target with a benchmark value or an average value, and calculate the relative analysis efficiency. The relative analysis efficiency can reflect the performance of each parsing target during the analysis process. Adjust the initial time aggregation granularity through the relative analysis efficiency. Different relative analysis efficiencies correspond to different adjustment coefficients, and the adjustment is made by multiplying the adjustment coefficient by the initial time aggregation granularity.

[0029] Step S102, perform multi-level surface information time aggregation on the log to be analyzed according to the time aggregation granularity, and generate multiple aggregated surface information sets, each of which corresponds to a deep information parsing target.

[0030] In some embodiments of the present application, perform multi-level surface information time aggregation on the log to be analyzed according to the time aggregation granularity, and generate multiple aggregated surface information sets, including Perform surface information time aggregation with the corresponding time aggregation granularity on the surface information on the log to be analyzed through the matching relationship between the surface information and the deep information parsing target, map the time aggregation granularity to the time window size, and perform the aggregation operation of the surface information within the time window to obtain multiple aggregated surface information sets at multiple levels.

[0031] In this embodiment, according to the analysis requirements, different levels of time aggregation granularity are determined, such as second level, minute level, hour level, day level, etc. Each time aggregation granularity is mapped to a corresponding time window size. For example, a few - second - level aggregation may correspond to a time window of a few seconds, a few - minute - level aggregation may correspond to several time windows of 60 seconds, and so on. Within the time window, aggregation operations are performed on the matching surface information. Aggregation operations can include counting (such as the number of requests from a certain IP address within the time window), summing (such as the total access volume of a certain user within the time window), average value calculation (such as the average response time of a certain request type within the time window), etc. Each deep - information parsing target corresponds to an aggregated surface - information set respectively.

[0032] Step S103, extract surface events associated with the deep - information parsing target from each aggregated surface - information set, and generate deep events according to the surface events.

[0033] In this embodiment, for the surface events associated with the deep - information parsing target, in addition to the surface events under the matching relationship, other surface events are screened through event similarity to complement the matching relationship.

[0034] In some embodiments of the present application, extracting surface events associated with the deep - information parsing target from each aggregated surface - information set and generating deep events according to the surface events includes: denoting the surface events having a matching relationship with the deep - information parsing target as original surface events, denoting the surface events having no matching relationship with the deep - information parsing target as other surface events, calculating the event similarity between the original surface events and the other surface events, and using the event similarity to take some of the other surface events and all of the original surface events as the surface events associated with the deep - information parsing target;

[0035] Form an associated surface - event sequence according to the timestamps of the surface events associated with the deep - information parsing target respectively, take the associated surface - event sequence as the deep event, and describe the current state of the deep event through a Markov chain.

[0036] In this embodiment, the event similarity can be cosine similarity or the like. According to the calculation result of the event similarity and the analysis requirements, an association criterion is determined. For example, a similarity threshold can be set. When the similarity between other surface events and the original surface event exceeds this threshold, it is considered that they are associated with the deep information parsing target. According to the timestamps of the surface events associated with the deep information parsing target respectively, they are sorted and form an associated surface event sequence. The associated surface event sequence is used as the deep event. The deep event is a higher-level event representation, which contains multiple surface events associated with the deep information parsing target and their chronological order. In this embodiment, a Markov chain is selected to describe the current state of the deep event. A Markov chain is a stochastic process that assumes that the current state depends only on the previous state and is independent of earlier states. By constructing a Markov chain model, the state transition process of the deep event can be simulated and analyzed.

[0037] In some embodiments of the present application, before parsing the deep information parsing target according to the deep event, the method further includes Collect surface events for all possible states of each deep information parsing target, and construct a surface event sequence for each possible state of each deep information parsing target, and use a Markov chain to describe the surface event sequence in the possible state.

[0038] In this embodiment, all possible states of the parsing target are all possible states that may occur under this target. For example, all possible states of the user behavior pattern state include normal state, abnormal state, suspicious state, etc. For the possible states of each deep information parsing target, collect the surface events related to this state. These events can be obtained from various sources such as log data, sensor data, and user feedback. Sort out the collected surface events, including removing duplicate events, correcting error events, filling in missing events, etc., to ensure the accuracy and integrity of the event data. Sort the surface events in each possible state according to the timestamp or other appropriate sorting criteria to form an event sequence. Use the constructed Markov chain model to describe the surface event sequence in each possible state. By simulating the state transition process, the surface event sequence that may occur in the future can be predicted, or the rationality of the surface event sequence that has occurred can be analyzed.

[0039] Step S104, parse the deep information parsing target according to the deep event, so as to realize the in-depth mining and analysis of the log and optimize the system operation.

[0040] In this embodiment, the deep event is a time series of surface events, which is a relatively general and abstract concept. Analyzing this sequence to determine the specific state of the parsing target. Here, the analysis refers to calculating the similarity and state transition probability of the surface event sequence, so as to determine the specific state of the current parsing target reflected by the log.

[0041] In some embodiments of the present application, parsing the deep information parsing target according to the deep event includes calculating the similarity and state transition probability between the current state of each deep event and each possible state of the deep information parsing target, and determining the matching degree based on the similarity and state transition probability; Among them, is the matching degree between the j1-th current state and the j2-th possible state of the deep event, is the conversion coefficient between the j1-th current state and the j2-th possible state of the deep event, is the similarity between the j1-th current state and the j2-th possible state of the deep event, is the state transition probability between the j1-th current state and the j2-th possible state, is the constant between the j1-th current state and the j2-th possible state; Determine the current state of the deep event according to the matching degree between the current state of the deep event and each possible state of the deep information parsing target, so as to parse the deep information parsing target.

[0042] In this embodiment, calculate the similarity between the current state of each deep event and each possible state of the deep information parsing target. Here, the similarity includes two kinds of similarities. One is the similarity of surface events, and the other is the similarity between the orders of surface events. The similarity is determined by combining the two kinds of similarities. The state transition probability describes the probability of transitioning from the current state to each possible state. represents the correction of the similarity by the state transition probability. The greater the state transition probability, the higher the similarity correction. is a correction function size for balancing the state transition probability.

[0043] Correspondingly, the present application also provides a log parsing and recognition system, as Figure 2 shown, including, The first module is used to obtain the log to be analyzed and the deep information parsing target, identify the surface information and timestamp in the log to be analyzed, analyze the time-related requirements involved in the deep information parsing target, and determine the time aggregation granularity of each deep information parsing target according to the time-related requirements; The second module is used to perform multi-level surface information time aggregation on the log to be analyzed according to the time aggregation granularity, generate multiple aggregated surface information sets, and each aggregated surface information set corresponds to a deep information parsing target respectively; The third module is used to extract surface events associated with the deep information parsing target from each aggregated surface information set and generate deep events based on the surface events;

[0044] The fourth module is used to parse the deep information parsing target according to the deep events, so as to realize the in-depth mining and analysis of the log and optimize the system operation.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Analyze the time-related requirements involved in the deep information parsing target to determine the time requirement situations corresponding to different deep information parsing targets, thereby determining the time aggregation granularity, performing time aggregation on the surface information of the log. Through the time aggregation operation, the data redundancy and complexity of the surface information can be effectively reduced, providing a reliable basis for the subsequent analysis of deep information.

[0046] 2. Extract surface events associated with the deep information parsing target from each aggregated surface information set, generate deep events, parse the deep information parsing target according to the deep events, associate the aggregated surface events, use the time series of the associated surface events as the deep events, and parse the deep information parsing target by analyzing the time series of the surface events, improving the log parsing ability and adaptability, and helping the subsequent system optimization and monitoring.

[0047] Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by hardware or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various implementation scenarios of the present invention.

[0048] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the drawings are not necessarily essential for implementing the present invention.

[0049] Those skilled in the art can understand that the modules in the system in the implementation scenario can be distributed in the system of the implementation scenario according to the description of the implementation scenario, or can be correspondingly changed and located in one or more systems different from the present implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0050] As described above, it is only the preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes, shall be covered by the protection scope of the present invention.

Claims

1. A log parsing and identification method, characterized in that: include, Obtain the logs to be analyzed and the deep information analysis targets, identify the surface information and timestamps in the logs to be analyzed, analyze the time-related requirements involved in the deep information analysis targets, and determine the time aggregation granularity of each deep information analysis target based on the time-related requirements; Perform multi-level surface information time aggregation on the logs to be analyzed according to the time aggregation granularity to generate multiple aggregated surface information sets, each of which corresponds to a deep information analysis target; Extracting surface events associated with deep information analysis targets from each aggregated surface information set, and generating deep events based on the surface events; Analyze deep information analysis targets based on deep events to achieve in-depth mining and analysis of logs and optimize system operation.

2. The log parsing and identification method according to claim 1, characterized in that: Identify the surface information and timestamps in the logs to be analyzed, including: Determine the format of the log to be analyzed, extract the surface information in the log to be analyzed according to the format, and classify the surface information; Extract the timestamp from the log to be analyzed, convert the timestamp to a unified standard, and mark it on the corresponding surface information.

3. The log parsing and identification method according to claim 1, characterized in that: And analyze the time-related requirements involved in deep information analysis goals, include, Split each deep information analysis goal into multiple sub-goals, determine all behavioral events involved in each sub-goal based on surface information, determine the temporal activity of behavioral events based on the frequency and duration of behavioral events, integrate the temporal activity of all behavioral events involved in each sub-goal to determine the temporal activity of the sub-goal, thereby determining the time requirement level of each deep information analysis goal, and describe the time-related requirements involved in the deep information analysis goal through the time requirement level; in, is the time requirement level of the i1th deep information analysis target, is the conversion coefficient of the i1th deep information analysis target, n is the number of sub-targets of the i1th deep information analysis target, is the combined weight corresponding to the i2th sub-goal, is the temporal activity of the i2th sub-goal of the i1th deep information parsing goal, for The minimum value in for The maximum value in are the first constant and the second constant of the i1th deep information parsing target respectively, and [] represents the integer symbol.

4. The log parsing and identification method according to claim 3 is characterized in that: And determine the time aggregation granularity of each deep information analysis target based on time-related requirements, including: Determine an initial time aggregation granularity based on the time requirement level of each deep information analysis target, match the surface information with the deep information analysis target, determine the data output and resource allocation of the surface information corresponding to the deep information analysis target, generate relative analysis efficiency based on the data output and resource allocation, adjust the initial time aggregation granularity through the relative analysis efficiency, and obtain the time aggregation granularity of each deep information analysis target.

5. The log parsing and identification method according to claim 4 is characterized in that: Perform multi-level surface information time aggregation on the logs to be analyzed according to the time aggregation granularity to generate multiple aggregated surface information sets, including: Through the matching relationship between the surface information and the deep information analysis target, the surface information on the analyzed log is time aggregated at the corresponding time aggregation granularity, the time aggregation granularity is mapped to the time window size, and the surface information aggregation operation is performed within the time window to obtain multiple aggregated surface information sets at multiple levels.

6. The log parsing and identification method according to claim 4, characterized in that: Extract surface events associated with deep information analysis targets from each aggregated surface information set, and generate deep events based on surface events, including: Record the surface events that match the deep information analysis target as original surface events, record the surface events that do not match the deep information analysis target as other surface events, calculate the event similarity between the original surface events and other surface events, and use the event similarity to classify some other surface events and all original surface events as surface events associated with the deep information analysis target; According to the timestamps of the surface events associated with the deep information analysis target, a sequence of associated surface events is formed, and the sequence of associated surface events is used as the deep event, and the current state of the deep event is described by a Markov chain.

7. The log parsing and identification method according to claim 6, characterized in that: Before parsing the deep information parsing target according to the deep event, the method further includes: Surface events are collected for all possible states of each deep information analysis target, and a surface event sequence for each possible state of each deep information analysis target is constructed. The surface event sequence under possible states is described by a Markov chain.

8. The log parsing and identification method according to claim 7, characterized in that: Analyze deep information analysis targets based on deep events, include, Calculate the similarity and state transition probability between the current state of each deep event and each possible state of the deep information analysis target, and determine the matching degree based on the similarity and state transition probability; in, is the matching degree between the j1th current state and the j2th possible state of the deep event, is the conversion coefficient between the j1th current state and the j2th possible state of the deep event, is the similarity between the j1th current state and the j2th possible state of the deep event, is the state transition probability between the j1th current state and the j2th possible state, is the constant between the j1th current state and the j2th possible state; The current state of the deep event is determined according to the matching degree between the current state of the deep event and each possible state of the deep information parsing target, thereby parsing the deep information parsing target.

9. A log parsing and identification system, characterized in that: include, The first module is used to obtain the logs to be analyzed and the deep information analysis targets, identify the surface information and timestamps in the logs to be analyzed, analyze the time-related requirements involved in the deep information analysis targets, and determine the time aggregation granularity of each deep information analysis target based on the time-related requirements; The second module is used to perform multi-level surface information time aggregation on the log to be analyzed according to the time aggregation granularity, and generate multiple aggregated surface information sets, each of which corresponds to a deep information analysis target; The third module is used to extract surface events associated with the deep information analysis target in each aggregated surface information set, and generate deep events based on the surface events; The fourth module is used to parse deep information analysis targets based on deep events, so as to achieve deep mining and analysis of logs and optimize system operation.

Images