Log processing method and device

Abstract

The invention relates to a log processing method and device. The method comprises the following steps: acquiring an initial log data set of each service module in a service system; determining a log type of each initial log data set according to a target regular rule, a line feed and a separator data set; according to the log type, determining a parser matched with the initial log data set, and analyzing the initial log data set according to the parser to obtain a to-be-analyzed log data set; the to-be-analyzed log data set is used for determining the running state of the business system. By adopting the method, the accuracy of the log processing method can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a log processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] In modern information system operation and maintenance, logs are a crucial data source for recording system operating status, troubleshooting, and security auditing. To clearly understand the operational status of a business system, it is necessary to parse and analyze its logs to obtain analytical results.

[0003] In traditional technologies, a business system comprises various business modules. Each business module generates its own initial log dataset. Operations personnel then parse the log data within each initial log dataset to obtain a parsed initial log dataset. Finally, they analyze the parsed initial log dataset to determine the operational status of the business system.

[0004] However, in traditional techniques, the accuracy of manually parsing each initial log dataset based on subjective experience is low. Therefore, the accuracy of current log processing methods is relatively low. Summary of the Invention

[0005] Therefore, it is necessary to provide a log processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] Firstly, this application provides a log processing method, including:

[0007] Obtain the initial log dataset of each business module in the business system;

[0008] Based on the target regular expression rules, newline characters, and delimiter datasets, determine the log type for each of the initial log datasets;

[0009] Based on the log type, a parser matching the initial log dataset is determined, and the initial log dataset is parsed using the parser to obtain a log dataset to be analyzed; the log dataset to be analyzed is used to determine the operating status of the business system.

[0010] In one embodiment, obtaining the initial log dataset of each business module in the business system includes:

[0011] Collect the initial log dataset of each business module according to the timer of each business module in the business system.

[0012] In one embodiment, before collecting the initial log dataset of each business module according to its timer in the business system, the method further includes:

[0013] Based on the call frequency of each business module in the business system, determine the log collection cycle of the business module, and determine the timer of the business module based on the log collection cycle.

[0014] In one embodiment, determining the log type of each initial log dataset based on the target regular expression rule, newline character, and delimiter dataset includes:

[0015] For each initial log dataset, determine whether the initial log dataset contains a newline character;

[0016] If the initial log dataset does not contain the newline character, then the log type of the initial log dataset is determined according to the target regular expression rule and the delimiter dataset;

[0017] If the initial log dataset contains the newline character, then the log type of the initial log dataset is determined according to the target regular expression rule.

[0018] In one embodiment, the target regular expression rule and the first target regular expression rule and the second target regular expression rule, wherein determining the log type of the initial log dataset based on the target regular expression rule and the partitioned dataset includes:

[0019] If the initial log dataset conforms to the first target regular expression rule, the log type of the initial log dataset is determined to be the first type;

[0020] If the initial log dataset does not conform to the first target regular expression rule, determine whether the initial log dataset conforms to the second target regular expression rule based on the regular expression dataset.

[0021] If the initial log dataset conforms to the second target regular expression rule, the log type of the initial log dataset is determined to be the second type;

[0022] If the initial log dataset does not conform to the second target regular expression rule, the log type of the initial log dataset is determined based on the delimiter dataset.

[0023] In one embodiment, determining the log type of the initial log dataset based on the delimiter dataset includes:

[0024] Based on the delimiter dataset, determine whether a fixed delimiter exists in the initial log dataset;

[0025] If the fixed delimiter exists in the initial log dataset, then the log type of the initial log dataset is determined to be the third type;

[0026] If the fixed delimiter is not present in the initial log dataset, then the log type of the initial log dataset is determined to be the fourth type.

[0027] In one embodiment, determining the log type of the initial log dataset according to the target regular expression rule includes:

[0028] Based on the regular expression dataset, determine whether the initial log dataset satisfies the second target regular expression rule in the target regular expression rule;

[0029] If the initial log dataset conforms to the second target regular expression rule, the log type of the initial log dataset is determined to be the fifth type;

[0030] If the initial log dataset does not conform to the second target regular expression rule, the log type of the initial log dataset is determined to be the sixth type.

[0031] In one embodiment, the step of determining the parser matching the initial log dataset based on the log type, and parsing the initial log dataset according to the parser to obtain the log dataset to be analyzed, includes:

[0032] If the log type is the second type or the fifth type, the parser corresponding to the initial log dataset is determined to be a regular expression parser, and the fields of the initial log dataset are extracted according to the regular expression parser to obtain the log dataset to be analyzed;

[0033] If the log type is the third type, the parser corresponding to the initial log dataset is determined to be the delimiter parser, and the initial log dataset is segmented according to the delimiter parser to obtain the log dataset to be analyzed.

[0034] If the log type is the sixth type, the parser corresponding to the initial log dataset is determined to be a context recognition model, and the initial log dataset is merged according to the context recognition model to obtain the log dataset to be analyzed.

[0035] In one embodiment, the step of extracting fields from the initial log dataset using the regular expression parser to obtain the log dataset to be analyzed includes:

[0036] Based on the business module identifier and log type corresponding to the initial log dataset, filter the target regular expression template corresponding to the initial log dataset from each regular expression template;

[0037] Based on the target regular expression template and the regular expression parser, extract each target field from the initial log dataset, and construct the log dataset to be analyzed based on each target field.

[0038] Secondly, this application also provides a log processing apparatus, comprising:

[0039] The acquisition module is used to acquire the initial log dataset of each business module in the business system;

[0040] The determination module is used to determine the log type of each of the initial log datasets based on the target regular expression rules, newline characters, and delimiter datasets;

[0041] The parsing module is used to determine the parser that matches the initial log dataset according to the log type, and to parse the initial log dataset according to the parser to obtain the log dataset to be analyzed; the log dataset to be analyzed is used to determine the running status of the business system.

[0042] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0043] Obtain the initial log dataset of each business module in the business system;

[0044] Based on the target regular expression rules, newline characters, and delimiter datasets, determine the log type for each of the initial log datasets;

[0045] Based on the log type, a parser matching the initial log dataset is determined, and the initial log dataset is parsed using the parser to obtain a log dataset to be analyzed; the log dataset to be analyzed is used to determine the operating status of the business system.

[0046] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0047] Obtain the initial log dataset of each business module in the business system;

[0048] Based on the target regular expression rules, newline characters, and delimiter datasets, determine the log type for each of the initial log datasets;

[0049] Based on the log type, a parser matching the initial log dataset is determined, and the initial log dataset is parsed using the parser to obtain a log dataset to be analyzed; the log dataset to be analyzed is used to determine the operating status of the business system.

[0050] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0051] Obtain the initial log dataset of each business module in the business system;

[0052] Based on the target regular expression rules, newline characters, and delimiter datasets, determine the log type for each of the initial log datasets;

[0053] Based on the log type, a parser matching the initial log dataset is determined, and the initial log dataset is parsed using the parser to obtain a log dataset to be analyzed; the log dataset to be analyzed is used to determine the operating status of the business system.

[0054] The aforementioned log processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire initial log datasets from each business module in a business system; determine the log type of each initial log dataset based on target regular expression rules, newline characters, and delimiter datasets; determine the parser matching the initial log dataset based on the log type, and parse the initial log dataset using the parser to obtain a log dataset to be analyzed; the log dataset to be analyzed is used to determine the operating status of the business system. This method, by using target regular expression rules, newline characters, and delimiter datasets, can quickly determine the log type of the initial log dataset, thereby enabling parsing of the initial log dataset using a parser matching the log type. This avoids human intervention, achieving automated parsing of the initial log dataset, improving the accuracy of parsing the initial log dataset, and thus improving the accuracy of the log processing method. Simultaneously, this log processing method can automatically parse the initial log dataset, improving log processing efficiency. Furthermore, by setting a timer based on the call frequency of the business modules and collecting the initial log datasets of the business modules according to the timer, the data volume of the initial log dataset remains stable, avoiding congestion in the parsing process due to a large initial log dataset volume, and improving log processing efficiency. In addition, the parser can be customized according to the log type of the initial log dataset of the business system, making the log processing method applicable to various business systems and expanding the scope of application of the log processing method. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a log processing method in one embodiment;

[0057] Figure 2 This is a flowchart illustrating the process of determining the log type based on a newline character in one embodiment.

[0058] Figure 3 This is a flowchart illustrating the process of determining the log type based on a target regular expression rule in one embodiment;

[0059] Figure 4 This is a schematic diagram of the process for determining the log type based on the delimiter dataset in one embodiment;

[0060] Figure 5 This is a flowchart illustrating the process of determining the log type based on a second target regular expression rule in one embodiment.

[0061] Figure 6 This is a schematic diagram of the process of parsing the log dataset to be analyzed in one embodiment;

[0062] Figure 7 This is a schematic diagram illustrating the process of parsing the log dataset to be analyzed using a regular expression parser in one embodiment.

[0063] Figure 8 This is an architecture diagram of a log processing system in an exemplary embodiment;

[0064] Figure 9 This is a structural block diagram of a log processing device in one embodiment;

[0065] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0068] In modern information system operation and maintenance, logs are a crucial data source for recording system operating status, troubleshooting, and security auditing. Their structured processing is a key step in achieving automated monitoring, intelligent analysis, and the construction of log management platforms. To clearly understand the operational status of business systems, it is necessary to parse and analyze the logs to obtain analytical results.

[0069] In traditional technologies, a business system comprises various business modules. Each business module generates its own initial log dataset. Operations personnel then parse the log data within each initial log dataset to obtain a parsed initial log dataset. Finally, they analyze the parsed initial log dataset to determine the operational status of the business system.

[0070] However, in traditional techniques, the accuracy of manually parsing each initial log dataset based on subjective experience is low. Therefore, the accuracy of current log processing methods is relatively low.

[0071] Furthermore, in practical applications, logs are diverse in type and complex in format, and the following problems also exist:

[0072] The diversity of log formats makes structuring difficult: existing tools typically require configuring separate parsing rules for each log format, lacking universality and automatic recognition capabilities;

[0073] Multi-line logs are difficult to process efficiently: Multi-line logs output by some business systems (such as Java exception stack traces and database transaction logs) cannot be accurately merged, affecting the accuracy of subsequent analysis;

[0074] Lack of standardized processing procedures: Different types of log processing procedures are not uniform, which increases operation and maintenance costs and the probability of errors.

[0075] To address the aforementioned technical issues, this application provides a log processing method. By using target regular expression rules, newline characters, and delimiter datasets, the log type of the initial log dataset can be quickly determined. This allows for the parsing of the initial log dataset using a parser that matches the log type, avoiding manual intervention and achieving automated parsing of the initial log dataset. This improves the accuracy of parsing the initial log dataset and consequently enhances the accuracy of the log processing method. Furthermore, this application employs a standard processing flow, reducing operational costs and the probability of errors.

[0076] Furthermore, this application, through log type and parser, can automatically parse various types of initial log datasets, avoiding the need to configure separate parsing rules for each type (format) of initial log dataset. Moreover, this application adopts a modular, plug-in structure, facilitating subsequent expansion to new log types and novelties, thus improving the sustainable evolution capability and versatility of the log processing method.

[0077] In addition, when the log type of the initial log dataset is multi-line, this application uses a context recognition model to merge multi-line logs, which solves the problems of errors in merging multi-line logs and loss of key information in traditional methods, and improves the accuracy of log processing.

[0078] In one embodiment, such as Figure 1 As shown, a log processing method is provided. This application embodiment uses the application of this method to a computer device as an example for illustration. This application embodiment does not limit the execution device of a log processing method, and includes the following steps 102 to 106:

[0079] Step 102: Obtain the initial log dataset of each business module in the business system.

[0080] In implementation, the computer equipment collects initial log data for each business module in the business system, according to the timer corresponding to that business module. The timer is set based on the calling frequency of the business module.

[0081] Specifically, the business system contains different business modules, each with a different function. To perform a comprehensive analysis of the business system, it is necessary to obtain the initial log datasets for the entire system. Each business module in the business system has a different call frequency. To accommodate the call frequency of each business module, the computer device sets a timer for each module based on its call frequency. The higher the call frequency, the shorter the timer period. Then, the computer device obtains the initial log dataset for that business module according to its timer. The initial log dataset is the set of business data generated by the business module during its operation.

[0082] In an exemplary embodiment, the business system is a banking business system. The banking business system includes a transfer module, a loan module, and a financial product module. The computer device sets a first timer for the transfer module based on its call frequency, and sets a second timer for the loan module based on its call frequency. Simultaneously, the computer device sets a third timer for the financial product module based on its call frequency. Based on the first timer, the initial log dataset for the transfer module is obtained, and based on the second timer, the initial log dataset for the loan module is obtained. Then, the computer device obtains the initial log dataset for the financial product module based on the third timer.

[0083] Step 104: Determine the log type for each initial log dataset based on the target regular expression rules, newline character, and delimiter dataset.

[0084] In implementation, for each initial log dataset, the computer device determines whether the initial log dataset contains a newline character, obtaining a first determination result. Then, based on the first determination result, the target regular expression rule, and the delimiter dataset, the computer device determines the log type of that initial log dataset.

[0085] Specifically, due to the different execution logic of each business module, the initial log datasets generated by each business module have different formats. To unify the analysis of the initial log datasets, they need to be parsed according to their format. Therefore, for each initial log dataset, the computer device determines whether it contains multiple lines of logs based on the newline character, obtaining a first determination result. Then, based on the first determination result, the target regular expression rule, and the delimiter dataset, the computer device determines the log type of the initial log dataset.

[0086] Step 106: Determine the parser that matches the initial log dataset based on the log type, and parse the initial log dataset according to the parser to obtain the log dataset to be analyzed.

[0087] The log dataset to be analyzed is used to determine the operating status of the business system.

[0088] In implementation, the computer device determines the parser that matches the initial log dataset based on the log type and the correspondence between each log type and each parser. Then, the computer device calls the parser to parse the initial dataset, obtaining the parsed initial log dataset, and determines the parsed initial log dataset as the dataset to be analyzed.

[0089] Specifically, log types are divided into six types: Type 1, Type 2, Type 3, Type 4, Type 5, and Type 6. Type 1 is JSON. Type 2 is a single-line regular expression type. Type 3 is a delimiter type. Type 4 is a single-line type. Type 5 is a multi-line regular expression type. Type 6 is a multi-line type. If the initial log dataset is of type 1, the computer device determines the parser corresponding to the initial log dataset as a JSON parser. The computer device performs format conversion on the initial log dataset based on the JSON parser to obtain the log dataset to be analyzed. If the log type is type 2 or type 5, the computer device determines the parser corresponding to the initial log dataset as a regular expression parser and extracts the target fields from the initial log dataset according to the regular expression parser to obtain the log dataset to be analyzed. If the log type is type 3, the computer device determines the parser corresponding to the initial log dataset as a delimiter parser and segments the initial log dataset according to the delimiter parser to obtain the log dataset to be analyzed. If the log type is the sixth type, the computer device determines that the parser corresponding to the initial log dataset is the context recognition model, and performs merging processing on the initial log dataset according to the context recognition model to obtain the log dataset to be analyzed.

[0090] In one optional embodiment, the computer device analyzes the log dataset to be analyzed based on a log analysis program to obtain analysis results. These analysis results can characterize whether the business modules of the business system are operating normally.

[0091] In the above log processing method, the log type of the initial log dataset can be quickly determined by using target regular expression rules, newline characters, and delimiter datasets. Then, the initial log dataset can be parsed and processed by a parser that matches the log type, avoiding human intervention and realizing automated parsing of the initial log dataset. This improves the accuracy of parsing the initial log dataset and, consequently, the accuracy of the log processing method.

[0092] In an exemplary embodiment, the specific processing procedure of step 102 includes: collecting the initial log dataset of each business module according to the timer of each business module in the business system.

[0093] In implementation, for each business module in the business system, when the timer of the business module is triggered at the current time, the computer equipment collects the initial log dataset of that business module through the collector.

[0094] Specifically, a business module may contain different business sub-modules, and each sub-module will generate logs. When the timer of a business module is triggered at the current time, the computer device retrieves the log data generated by each business sub-module within the business module for each business module in the business system, and constructs the initial log dataset for that business module based on the log data.

[0095] In an exemplary embodiment, taking a transfer module as an example, the transfer module includes an identity verification submodule, an information entry submodule, a balance check submodule, a path selection submodule, a processing submodule, a transaction fee calculation submodule, a result feedback submodule, and a query submodule. The computer device obtains log data from the identity verification submodule, information entry submodule, balance check submodule, path selection submodule, processing submodule, transaction fee calculation submodule, result feedback submodule, and query submodule, and constructs an initial log dataset for the transfer module based on the log data.

[0096] Optionally, the format of the initial log dataset may be, but is not limited to, data from a file, network stream, or message queue. The collector may be, but is not limited to, Filebeat (a lightweight collector), Logstash (a collector), or Kafka (a distributed stream processing platform).

[0097] In this embodiment, the initial log datasets of each business module in the business system are collected periodically by a collector and a timer, so that the collection frequency is adapted to each business module, which facilitates the subsequent parsing of the initial log datasets.

[0098] In an exemplary embodiment, before collecting the initial log dataset based on a timer, the timer for the business module also needs to be determined. Therefore, the specific processing procedure of this log processing method further includes:

[0099] Based on the call frequency of each business module in the business system, determine the log collection cycle of the business module, and based on the log collection cycle, determine the timer of the business module.

[0100] In implementation, the computer equipment determines the log collection cycle for each business module in the business system based on the module's call frequency and the correspondence between call frequency ranges and collection cycles. Then, the computer equipment sets the timer for the business module using the log collection cycle as a parameter. Higher call frequencies result in shorter log collection cycles and shorter time differences between timers.

[0101] Specifically, the call frequency refers to the operating frequency of the business module. If a business module operates more frequently, its call frequency is higher. The computer equipment pre-sets the correspondence between each call frequency range and each data collection period. For each business module in the business system, the computer equipment determines the target call frequency range within each call frequency range and sets the corresponding data collection period as the log collection period for that business module. Then, the computer equipment sets the timer for the business module based on the log collection period as the time difference.

[0102] For example, taking a banking system as an example, the transfer module operates more frequently than the financial product system. That is, the transfer module is called more often than the financial product module. The correspondence between different call frequency ranges and collection periods is as follows: a collection period of 14 days corresponds to a call frequency of (0, 10 times / second); a collection period of 7 days corresponds to a call frequency of (10, 50 times / second); and a collection period of 1 day corresponds to a call frequency of (50, +∞ times / second). If the transfer module's call frequency is 70 times / second, the computer determines the log collection period for the transfer module to be 1 day and sets the timer for the transfer module based on a 1-day parameter. If the financial product module's call frequency is 40 times / second, the computer determines the log collection period for the financial product module to be 7 days.

[0103] In this embodiment, the timer of the business module is determined by the calling frequency of the business module, and the initial log dataset of the business module is collected based on the timer. This allows the collection frequency to be adapted to each business module, keeping the amount of data in the initial log dataset in a stable range, which is convenient for subsequent parsing of the initial log dataset.

[0104] In one exemplary embodiment, such as Figure 2 As shown, the specific processing procedure of step 104 includes steps 202 to 206. Wherein:

[0105] Step 202: For each initial log dataset, determine whether the initial log dataset contains a newline character.

[0106] In implementation, for each initial log dataset, the computer device determines whether the initial log dataset contains a newline character, thereby determining whether the initial log dataset contains multiple lines of logs. If the initial log dataset does not contain a newline character, the computer device performs step 204 below; if the initial log dataset contains a newline character, the computer device performs step 206 below; if the initial log dataset does not contain a newline character, the computer device performs step 204 below.

[0107] Specifically, since the processing rules for single-line and multi-line logs differ, the computer device needs to determine whether the initial log dataset is in single-line or multi-line format. For each initial log dataset, the computer device determines whether it contains a newline character. If the initial log dataset contains a newline character, the computer device determines that the initial log dataset is in multi-line format and proceeds to step 206 to determine whether the initial log dataset conforms to the target regular expression rule. If the initial log dataset does not contain a newline character, the computer device determines that the initial log dataset is in single-line format and proceeds to step 204 to determine whether the initial log dataset conforms to the target regular expression rule or the delimiter.

[0108] Step 204: If the initial log dataset does not contain newline characters, determine the log type of the initial log dataset based on the target regular expression rule and the delimiter dataset.

[0109] The target regularity rule includes a first target regularity rule and a second target regularity rule.

[0110] In implementation, if the initial log dataset does not contain newline characters, the computer device determines whether the initial log dataset conforms to the first target regular expression rule, obtaining a second determination result. Then, based on the second determination result, the second target regular expression rule, and the delimiter dataset, the computer device determines the log type of the initial log dataset.

[0111] Specifically, if the second judgment result indicates that the initial log dataset conforms to the first target regular expression rule, then the computer device determines that the log type of the initial log dataset is the first type. Here, the first target regular expression rule is a JSON (JavaScript Object Notation) rule, and the first type is the JSON type. If the second judgment result indicates that the initial log dataset does not conform to the first target regular expression rule, then the computer device determines whether the initial log dataset conforms to the second target regular expression rule, thereby determining whether the initial log dataset is a regularized log dataset, and obtaining the third judgment result. Then, the computer device determines the log type of the initial log dataset based on the third judgment result and the delimiter dataset.

[0112] Step 206: If the initial log dataset contains newline characters, determine the log type of the initial log dataset according to the target regular expression rule.

[0113] Among them, the target regularity rule includes the second target regularity rule.

[0114] In implementation, if the initial log dataset contains newline characters, the computer device determines that the initial log dataset contains multiple lines of logs. Then, the computer device determines whether the initial log dataset conforms to the second objective regular expression rule, obtains the fourth judgment result, and determines the log type of the initial log dataset based on the fourth judgment result.

[0115] Specifically, if the fourth judgment result indicates that the initial log dataset conforms to the second target regular expression rule, then the computer device determines that the log type of the initial log dataset is a multi-line regular expression type, i.e., the fourth type. If the fourth judgment result indicates that the initial log dataset does not conform to the second target regular expression rule, then the computer device determines that the log type of the initial log dataset is the sixth type.

[0116] In this embodiment, by determining whether the initial log dataset contains newline characters, it is possible to distinguish whether the initial log dataset is a single-line log or a multi-line log, thus achieving a preliminary classification of the initial log dataset.

[0117] In an exemplary embodiment, the target regular expression rule and the regular expression rule containing the first target regular expression rule and the second target regular expression rule are, as follows: Figure 3 As shown, the specific processing procedure of step 204 includes steps 302 to 308. Wherein:

[0118] Step 302: If the initial log dataset conforms to the first target regular expression rule, determine that the log type of the initial log dataset is the first type.

[0119] The first target regularization rule is a JSON rule.

[0120] In implementation, if the initial log dataset conforms to the first target regularity rule, the computer device determines that the log type of the initial log dataset is the first type, namely Json type, and determines that the initial log dataset is a Json log (JsonLog).

[0121] Step 304: If the initial log dataset does not conform to the first target regular expression rule, determine whether the initial log dataset conforms to the second target regular expression rule based on the regular expression dataset.

[0122] The second objective regularization rule contains a regularization dataset. This dataset contains various regularization rules.

[0123] In implementation, if the initial log dataset does not conform to the first target regularity rule, the computer device determines whether the initial log dataset conforms to the second target regularity rule, thereby determining whether the initial log dataset is a regular log dataset. If the initial log dataset conforms to the second target regularity rule, the computer device executes step 306 below. If the initial log dataset does not conform to the second target regularity rule, the computer device executes step 308 below.

[0124] Specifically, if the initial log dataset does not conform to the first target regular expression rule, the computer device determines whether the initial dataset matches the regular expression data for each regular expression data in the regular expression dataset, thereby determining whether the initial log dataset conforms to the second target regular expression rule. If the initial log dataset conforms to the second regular expression rule, the computer device determines that the log type of the initial log dataset is a single-line regular expression type. If the initial log dataset does not conform to the second regular expression rule, the computer device determines the log type of the initial log dataset based on the delimiter dataset.

[0125] Optionally, the second target regular expression rule can be a common regular expression rule. The second target regular expression rule can be, but is not limited to, a system regular expression rule or a database log regular expression rule. This application embodiment does not limit the second target regular expression rule according to the needs of the business system.

[0126] Step 306: If the initial log dataset conforms to the second target regularity rule, determine that the log type of the initial log dataset is the second type.

[0127] In implementation, if the initial log dataset conforms to the second target regular expression rule, the computer device determines that the log type of the initial log dataset is the second type. The second type is a single-line regular expression type. That is, the computer device determines that the initial log dataset is a Regex-based Single-line Log.

[0128] Step 308: If the initial log dataset does not conform to the second target regular expression rule, determine the log type of the initial log dataset based on the delimiter dataset.

[0129] In practice, if the initial log dataset does not conform to the second regular expression rule, the computer device determines whether the initial log dataset has a fixed delimiter based on the delimiter dataset, thereby determining the log type of the initial log dataset.

[0130] Optionally, the delimiter data may include, but is not limited to, spaces, commas, and tabs. This application embodiment does not limit the delimiters included in the delimiter data.

[0131] In this embodiment, by judging whether the initial log dataset conforms to the first target regular expression rule and the second target regular expression rule, it is possible to quickly determine whether the initial log dataset is a JSON log or a single-line regular expression log, thus realizing the automatic determination of the log type.

[0132] In one exemplary embodiment, such as Figure 4 As shown, the specific processing procedure of step 308 includes steps 402 to 406. Wherein:

[0133] Step 402: Based on the delimiter dataset, determine whether a fixed delimiter exists in the initial log dataset.

[0134] In implementation, for each delimiter in the delimiter dataset, the computer device determines whether the delimiter exists at a fixed position in the initial log dataset. If the delimiter exists at the fixed position, the computer device determines that the delimiter is a fixed delimiter. If the delimiter does not exist at the fixed position, the computer device determines that the delimiter is a non-fixed delimiter.

[0135] Step 404: If a fixed delimiter exists in the initial log dataset, then the log type of the initial log dataset is determined to be the third type.

[0136] In implementation, if a fixed delimiter exists in the initial log dataset, the computer device determines the initial log type to be the third type. The third type is the delimiter type. That is, the computer device determines the initial log dataset to be a delimited log.

[0137] Step 406: If there is no fixed delimiter in the initial log dataset, then the log type of the initial log dataset is determined to be the fourth type.

[0138] In implementation, if the initial log dataset contains logs without a fixed delimiter, the computer device determines the log type of the initial log dataset to be type four. Type four is a single-line type. That is, the computer device determines the initial log dataset as a single-line log.

[0139] In this embodiment, by determining whether there is a fixed delimiter in the initial log dataset, the log type of the initial log dataset can be directly determined. This enables automatic determination of the type based on the content characteristics of the initial log dataset, allowing the corresponding parsing process to be started without manual configuration, thus improving log processing efficiency.

[0140] In one exemplary embodiment, such as Figure 5 As shown, the specific processing procedure of step 206 includes steps 502 to 506. Wherein:

[0141] Step 502: Determine whether the initial log dataset satisfies the second target regular expression rule in the target regular expression rule based on the regular expression dataset.

[0142] The target regular expression rule contains a second target regular expression rule, and the second target regular expression rule contains a regular expression dataset. The regular expression dataset contains various regular expression data.

[0143] In implementation, the computer device determines whether the log data in the initial log dataset matches each regular expression rule. If the initial log dataset matches all regular expression rules, the computer device determines that the initial log dataset satisfies the second target regular expression rule. If there are initial log datasets that do not match the target regular expression rules, the computer device determines that the initial log dataset does not conform to the second target regular expression rule.

[0144] In one exemplary embodiment, an access log rule is used as an example of a second target regular expression rule. The regular expression data in this second target regular expression rule are IP (Internet Protocol) regular expression data, URL (Uniform Resource Locator) regular expression data, request method regular expression data, and response status code request data, respectively. The computer device extracts IP addresses from the initial log dataset based on the IP regular expression data. If the IP address is successfully extracted, the computer device determines that the initial log dataset matches the IP regular expression data. If IP address extraction fails, the computer device determines that the initial log dataset does not match the IP regular expression data. The computer device extracts URL addresses from the initial log dataset based on the URL regular expression data. If the URL address is successfully extracted, the computer device determines that the initial log dataset matches the URL regular expression data. If URL address extraction fails, the computer device determines that the initial log dataset does not match the URL regular expression data. The computer device extracts request method addresses from the initial log dataset based on the request method regular expression data. If the request method address is successfully extracted, the computer device determines that the initial log dataset matches the request method regular expression data. If extracting the request method address fails, the computer device determines that the initial log dataset does not match the request method regular expression data. The computer device extracts the response status code address from the initial log dataset based on the response status code regular expression data. If the response status code address is successfully extracted, the computer device determines that the initial log dataset matches the response status code regular expression data. If extracting the response status code address fails, the computer device determines that the initial log dataset does not match the response status code regular expression data. If the initial log dataset matches the full regular expression data, the computer device determines that the initial log dataset satisfies the second target regular expression rule. If there are instances where the initial log dataset does not match the target regular expression data, the computer device determines that the initial log dataset does not conform to the second target regular expression rule.

[0145] Optionally, the regular expression rule set may include, but is not limited to, IP regular expression data, timestamp data, and HTTP (Hypertext Transfer Protocol) regular expression data, as determined by business requirements. This application embodiment limits the content of the regular expression rule set.

[0146] Step 504: If the initial log dataset conforms to the second objective regularity rule, determine that the log type of the initial log dataset is the fifth type.

[0147] In implementation, if the initial log dataset conforms to the second objective regular expression rule, the computer device determines the log type of the initial log dataset to be the fifth type. The fifth type is a multi-line regular expression type. That is, the computer device determines the initial log dataset as a Regex-based Multi-line Log.

[0148] Step 506: If the initial log dataset does not conform to the second target regularity rule, determine the log type of the initial log dataset as the sixth type.

[0149] In implementation, if the initial log dataset does not conform to the second objective regularization rule, the computer device determines the log type of the initial log dataset to be the sixth type. The fifth type is a multi-line type. That is, the computer device determines the initial log dataset to be a multi-line log.

[0150] In an alternative embodiment, the computer device detects logical relationships between log lines to determine whether the initial log dataset is a specific multi-line log type (e.g., whether the previous line ends with a specific character).

[0151] In this embodiment, the log type of the initial log dataset is determined by detecting whether it conforms to the second target regular expression rule. This facilitates the subsequent matching of the parser based on the log type and the parsing of the initial log dataset by the parser, thereby achieving automated parsing of the initial log dataset and improving parsing efficiency.

[0152] In one exemplary embodiment, such as Figure 6 As shown, the specific processing procedure of step 106 includes steps 602 to 606. Wherein:

[0153] Step 602: If the log type is type 2 or type 5, determine that the matching parser corresponding to the initial log dataset is a regular expression parser, and extract fields from the initial log dataset according to the regular expression parser to obtain the log dataset to be analyzed.

[0154] In implementation, if the log type is type two or five, the computer device determines the initial log dataset as a regular expression dataset. Then, the computer device identifies the regular expression parser that matches the initial log dataset. Next, based on the business information corresponding to the initial log dataset and the regular expression parser, the computer device extracts fields from the initial log dataset to obtain the log dataset to be analyzed.

[0155] Specifically, the computer equipment has pre-set regular expression templates. If the log type is type two or type five, the computer equipment determines the initial log dataset as a regular expression dataset. Based on the business information and log type corresponding to the initial log dataset, the computer equipment determines the target regular expression template corresponding to the initial log dataset from among the various regular expression templates. Then, based on the target regular expression template and the regular expression parser, the computer equipment extracts the target fields from the initial log dataset to obtain the log dataset to be analyzed.

[0156] Step 604: If the log type is the third type, determine that the parser corresponding to the initial log dataset is the delimiter parser, and perform segmentation processing on the initial log dataset according to the delimiter parser to obtain the log dataset to be analyzed.

[0157] In implementation, if the log type is type 3, the computer device will determine the pre-set delimiter parser as the parser corresponding to the initial log dataset. Then, based on the delimiter parser and the fixed delimiters in the initial log dataset, the computer device will slice the initial log dataset and map each sliced ​​log data into key-value pairs to obtain the initial dataset to be analyzed. Finally, the computer device will perform format conversion on the initial dataset to be analyzed according to a preset output format to obtain the dataset to be analyzed.

[0158] Specifically, the output format is JSON. If the log type is type 3, the computer device determines the initial log dataset to be a delimited log. The computer device identifies the delimiter parser for parsing the delimited log as the parser corresponding to the initial log dataset, and segments the initial log dataset according to the delimiter parser and the fixed delimiters in the initial log dataset to obtain individual log data. Then, the computer device maps each segmented log data into key-value pairs to obtain the initial dataset to be analyzed, and performs format conversion on the initial dataset to be analyzed according to the preset JSON format to obtain the dataset to be analyzed.

[0159] Step 606: If the log type is the sixth type, determine that the parser corresponding to the initial log dataset is the context recognition model, and merge the initial log dataset according to the context recognition model to obtain the log dataset to be analyzed.

[0160] In implementation, if the log type is type six, the computer device determines the context recognition model as the parser corresponding to the initial log dataset. Then, the computer device inputs the initial log dataset into the context recognition model, which performs merging processing on the initial log dataset to obtain the log dataset to be analyzed.

[0161] Specifically, if the log type is type 6, the computer device determines that the initial log dataset is a multi-line log and identifies the context-aware algorithm as the parser corresponding to the initial log dataset. Then, the computer device inputs the initial log dataset into the context-aware algorithm, which merges multiple physical lines in the initial log dataset into a single logical log event and outputs it in JSON format to obtain the log dataset to be analyzed.

[0162] In this embodiment, the initial log dataset is parsed according to the log type and the parser corresponding to the initial log dataset, avoiding manual intervention and achieving automated parsing of the initial log dataset. This improves the accuracy of parsing the initial log dataset and, consequently, the accuracy of the log processing method. Furthermore, the initial log dataset of this application supports multiple log types, making it suitable for different application scenarios and system environments. Additionally, the unified processing flow simplifies the configuration difficulty of log collection and analysis. This application uses a context-aware model to process multi-line context-aware, multi-type initial log datasets, solving the problems of multi-line log merging errors and loss of key information in traditional methods, thus improving the accuracy of log processing.

[0163] In one exemplary embodiment, such as Figure 7 As shown, the specific processing steps in step 602, which involve extracting fields from the initial log dataset using a regular expression parser to obtain the log dataset to be analyzed, include steps 702 to 704. Wherein:

[0164] Step 702: Based on the business module identifier and log type corresponding to the initial log dataset, filter the target regular expression template corresponding to the initial log dataset from each regular expression template.

[0165] Log types are divided into the third type and the fifth type.

[0166] In implementation, if the log type is type three, the computer device filters each single-line regular expression template from the various regular expression templates, and then filters the target regular expression template corresponding to the initial log dataset from the various single-line regular expression templates based on the business module identifier of the initial log dataset. If the log type is type five, the computer device filters each multi-line regular expression template from the various regular expression templates, and then filters the target regular expression template corresponding to the initial log dataset from the various multi-line regular expression templates based on the business module identifier of the initial log dataset.

[0167] In an optional embodiment, if no target regular expression template corresponding to the initial log dataset is found, the computer device receives a new regular expression template uploaded by the user and identifies this new regular expression template as the target regular expression template. The computer device will preferentially parse the initial log dataset based on this target regular expression template, improving the flexibility and adaptability of the log processing method.

[0168] Step 704: Extract each target field from the initial log dataset based on the target regular expression template and the regular expression parser, and construct the log dataset to be analyzed based on each target field.

[0169] In practice, the computer equipment extracts each target field from the initial log dataset based on the target regular expression template and the regular expression parser, and combines the target fields to obtain the log dataset to be analyzed.

[0170] Specifically, if the initial log dataset has a log type of type 3, the computer device parses the initial log dataset based on a regular expression parser and the regular expression parsing template in the target regular expression template, and extracts the target fields from the parsed initial log dataset. Then, the computer device combines the target fields to obtain the log dataset to be analyzed. If the initial log dataset has a log type of type 5, the computer device merges the logs in the initial log dataset based on a context recognition model, obtaining a merged initial log dataset. Then, the computer device parses the initial log dataset based on a regular expression parser and the regular expression parsing template in the target regular expression template, and extracts the target fields from the parsed initial log dataset. Then, the computer device combines the target fields to obtain the log dataset to be analyzed. It should be noted that the computer device outputs the log dataset to be analyzed in JSON format.

[0171] In this embodiment, by using log type and business module identifier, a target regular expression template that matches the initial log dataset can be automatically determined. This template is then used to automatically parse the initial log dataset, yielding the dataset to be analyzed. Furthermore, the log fields in the dataset to be analyzed are clearly defined, facilitating subsequent retrieval, aggregation, and visualization.

[0172] In one exemplary embodiment, Figure 8 This is an architecture diagram of a log processing system in an exemplary embodiment. Figure 8 As shown, the log processing system includes a log input module, a log type recognition module, a formatting engine module, and an output module. Among them:

[0173] The log input module receives raw log data streams. It collects initial log datasets from the business system using collectors such as Filebeat, Logstash, and Kafka.

[0174] The log type identification module is used to determine whether a log belongs to a single-line, multi-line, single-line regular expression, multi-line regular expression, JSON, or delimiter log based on the characteristics of the log content.

[0175] The formatting engine module is used to call the corresponding parser for structured processing based on the recognition results. This formatting engine includes single-line log processing, multi-line log processing, single-line regular expression processing, multi-line regular expression processing, JSON type log processing, and delimiter log processing engines. The formatting engine adopts a modular design, which allows for the rapid integration of new log parsing capabilities.

[0176] The output module is used to output the structured logs to downstream systems (e.g., Clickhouse, a database management system) in standard formats such as JSON.

[0177] Optionally, the log processing system also includes a performance optimization module, which performs optimization measures such as caching, batch processing, and compressed transmission of high-frequency logs to improve overall throughput.

[0178] If only the log input module, formatting processing engine module, and output module are used, it is suitable for scenarios that only need to identify the log type and output the classification results, such as for log classification statistics or preliminary screening.

[0179] If the full modules of the log processing system are used, they have complete recognition and structuring capabilities, and are suitable for most log processing needs.

[0180] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0181] Based on the same inventive concept, this application also provides a log processing apparatus for implementing the log processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more log processing apparatus embodiments provided below can be found in the limitations of the log processing method described above, and will not be repeated here.

[0182] In one exemplary embodiment, such as Figure 9 As shown, a log processing device 900 is provided, including: an acquisition module 901, a determination module 902, and a parsing module 903, wherein:

[0183] Module 901 is used to obtain the initial log dataset of each business module in the business system.

[0184] The determination module 902 is used to determine the log type of each initial log dataset based on the target regular expression rules, newline characters, and delimiter dataset.

[0185] The parsing module 903 is used to determine the parser that matches the initial log dataset based on the log type, and to parse the initial log dataset according to the parser to obtain the log dataset to be analyzed; the log dataset to be analyzed is used to determine the running status of the business system.

[0186] In an exemplary embodiment, the acquisition module 901 is specifically used to collect the initial log dataset of each business module in the business system according to the timer of each business module.

[0187] In one exemplary embodiment, the log processing apparatus 900 further includes:

[0188] The second determining module is used to determine the log collection cycle of the business module according to the call frequency of each business module in the business system, and to determine the timer of the business module according to the log collection cycle.

[0189] In one exemplary embodiment, the determining module 902 includes:

[0190] The first judgment submodule is used to determine whether each initial log dataset contains a newline character.

[0191] The first determination submodule is used to determine the log type of the initial log dataset based on the target regular expression rules and the delimiter dataset if the initial log dataset does not contain newline characters.

[0192] The second determination submodule is used to determine the log type of the initial log dataset according to the target regular expression rule if the initial log dataset contains newline characters.

[0193] In an exemplary embodiment, the target regularization rule and the first determining submodule, which includes a first target regularization rule and a second target regularization rule, are defined as follows:

[0194] The third determination submodule is used to determine the log type of the initial log dataset as the first type if the initial log dataset conforms to the first target regular expression rule.

[0195] The fourth determination submodule is used to determine whether the initial log dataset meets the second target regularity rule if the initial log dataset does not meet the first target regularity rule.

[0196] The fifth determination submodule is used to determine the log type of the initial log dataset as the second type if the initial log dataset conforms to the second target regularity rule.

[0197] The sixth determination submodule is used to determine the log type of the initial log dataset based on the delimiter dataset if the initial log dataset does not conform to the second target regularity rule.

[0198] In one exemplary embodiment, the sixth determining submodule includes:

[0199] The second judgment submodule is used to determine whether a fixed delimiter exists in the initial log dataset based on the delimiter dataset.

[0200] The seventh determination submodule is used to determine the log type of the initial log dataset as the third type if a fixed delimiter exists in the initial log dataset.

[0201] The eighth determination submodule is used to determine the log type of the initial log dataset as the fourth type if there is no fixed delimiter in the initial log dataset.

[0202] In one exemplary embodiment, the second determining submodule includes:

[0203] The third judgment submodule is used to determine whether the initial log dataset meets the second target regular expression rule in the target regular expression rule based on the regular expression dataset.

[0204] The ninth determination submodule is used to determine the log type of the initial log dataset as the fifth type if the initial log dataset conforms to the second objective regularity rule.

[0205] The tenth determination submodule is used to determine the log type of the initial log dataset as the sixth type if the initial log dataset does not conform to the second target regularity rule.

[0206] In one exemplary embodiment, the parsing module 903 includes:

[0207] The first extraction submodule is used to determine the matching parser corresponding to the initial log dataset as a regular expression parser if the log type is the second type or the fifth type, and to extract fields from the initial log dataset according to the regular expression parser to obtain the log dataset to be analyzed.

[0208] The segmentation module is used to determine the parser corresponding to the initial log dataset as the delimiter parser if the log type is the third type, and to segment the initial log dataset according to the delimiter parser to obtain the log dataset to be analyzed.

[0209] The merging submodule is used to determine that the parser corresponding to the initial log dataset is the context recognition model if the log type is the sixth type, and to merge the initial log dataset according to the context recognition model to obtain the log dataset to be analyzed.

[0210] In an exemplary embodiment, the first extraction submodule includes an eleventh determining submodule and a second extraction submodule. The second extraction submodule includes:

[0211] The twelfth determination submodule is used to filter the target regular expression template corresponding to the initial log dataset from each regular expression template based on the business module identifier and log type corresponding to the initial log dataset.

[0212] The third extraction submodule is used to extract each target field from the initial log dataset based on the target regular expression template and the regular expression parser, and to construct the log dataset to be analyzed based on each target field.

[0213] Each module in the aforementioned log processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0214] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a log processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0215] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0216] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0217] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0218] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0219] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0220] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0221] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A log processing method, characterized in that, The method includes: Obtain the initial log dataset of each business module in the business system; Based on the target regular expression rules, newline characters, and delimiter datasets, determine the log type for each of the initial log datasets; Based on the log type, a parser matching the initial log dataset is determined, and the initial log dataset is parsed using the parser to obtain a log dataset to be analyzed; the log dataset to be analyzed is used to determine the operating status of the business system.

2. The method according to claim 1, characterized in that, The process of obtaining the initial log dataset of each business module in the business system includes: Collect the initial log dataset of each business module according to the timer of each business module in the business system.

3. The method according to claim 2, characterized in that, Before collecting the initial log dataset of each business module according to its timer in the business system, the method further includes: Based on the call frequency of each business module in the business system, determine the log collection cycle of the business module, and determine the timer of the business module based on the log collection cycle.

4. The method according to claim 1, characterized in that, The step of determining the log type for each initial log dataset based on the target regular expression rules, newline characters, and delimiter dataset includes: For each initial log dataset, determine whether the initial log dataset contains a newline character; If the initial log dataset does not contain the newline character, then the log type of the initial log dataset is determined according to the target regular expression rule and the delimiter dataset; If the initial log dataset contains the newline character, then the log type of the initial log dataset is determined according to the target regular expression rule.

5. The method according to claim 4, characterized in that, The target regular expression rule and the first target regular expression rule and the second target regular expression rule, the step of determining the log type of the initial log dataset based on the target regular expression rule and the partitioned dataset, includes: If the initial log dataset conforms to the first target regular expression rule, the log type of the initial log dataset is determined to be the first type; If the initial log dataset does not conform to the first target regular expression rule, determine whether the initial log dataset conforms to the second target regular expression rule based on the regular expression dataset. If the initial log dataset conforms to the second target regular expression rule, the log type of the initial log dataset is determined to be the second type; If the initial log dataset does not conform to the second target regular expression rule, the log type of the initial log dataset is determined based on the delimiter dataset.

6. The method according to claim 5, characterized in that, The step of determining the log type of the initial log dataset based on the delimiter dataset includes: Based on the delimiter dataset, determine whether a fixed delimiter exists in the initial log dataset; If the fixed delimiter exists in the initial log dataset, then the log type of the initial log dataset is determined to be the third type; If the fixed delimiter is not present in the initial log dataset, then the log type of the initial log dataset is determined to be the fourth type.

7. The method according to claim 4, characterized in that, The step of determining the log type of the initial log dataset according to the target regular expression rule includes: Based on the regular expression dataset, determine whether the initial log dataset satisfies the second target regular expression rule in the target regular expression rule; If the initial log dataset conforms to the second target regular expression rule, the log type of the initial log dataset is determined to be the fifth type; If the initial log dataset does not conform to the second target regular expression rule, the log type of the initial log dataset is determined to be the sixth type.

8. The method according to claim 1, characterized in that, The step of determining the parser matching the initial log dataset based on the log type, and parsing the initial log dataset according to the parser to obtain the log dataset to be analyzed includes: If the log type is the second type or the fifth type, the parser corresponding to the initial log dataset is determined to be a regular expression parser, and the fields of the initial log dataset are extracted according to the regular expression parser to obtain the log dataset to be analyzed; If the log type is the third type, the parser corresponding to the initial log dataset is determined to be the delimiter parser, and the initial log dataset is segmented according to the delimiter parser to obtain the log dataset to be analyzed. If the log type is the sixth type, the parser corresponding to the initial log dataset is determined to be a context recognition model, and the initial log dataset is merged according to the context recognition model to obtain the log dataset to be analyzed.

9. The method according to claim 8, characterized in that, The step of extracting fields from the initial log dataset using the regular expression parser to obtain the log dataset to be analyzed includes: Based on the business module identifier and log type corresponding to the initial log dataset, filter the target regular expression template corresponding to the initial log dataset from each regular expression template; Based on the target regular expression template and the regular expression parser, extract each target field from the initial log dataset, and construct the log dataset to be analyzed based on each target field.

10. A log processing device, characterized in that, The device includes: The acquisition module is used to acquire the initial log dataset of each business module in the business system; The determination module is used to determine the log type of each of the initial log datasets based on the target regular expression rules, newline characters, and delimiter datasets; The parsing module is used to determine the parser that matches the initial log dataset according to the log type, and to parse the initial log dataset according to the parser to obtain the log dataset to be analyzed; the log dataset to be analyzed is used to determine the running status of the business system.

Images