A data processing method and a data processing device

Abstract

Embodiments of the present application disclose a data processing method and a data processing device, which are used for determining a log template, i.e. a parsing rule, of a log type corresponding to a plurality of logs of the same type according to the plurality of logs, and then extracting variables of the plurality of logs according to the log template to generate structured logs. That is, manual setting of the parsing rule is no longer needed, and manual maintenance and update of the parsing rule by a running process are avoided. The method comprises the following steps: a data processing device acquires a log set; the data processing device determines that N logs in the log set belong to a first type, wherein N is a positive integer; the data processing device determines a log template corresponding to the first type according to the N logs, wherein the log template corresponding to the first type is used to indicate variable positions of the N logs; and the data processing device extracts variables from one or more logs of the N logs according to the variable positions to generate structured logs.

Description

[0001] This application is a divisional application of the application filed with the China National Intellectual Property Administration on October 26, 2016, with application number 201610948580.6 and invention title "A Data Processing Method and Data Processing Device". Technical Field

[0002] This application relates to the field of computers, and more particularly to a data processing method and a data processing device. Background Technology

[0003] Log data is one of the most important data types in system operation and maintenance. Log data analysis is essential for website user behavior analysis and system operation and maintenance statistics. However, directly analyzing variables of a single log type from massive amounts of text logs is extremely difficult. For example, analyzing the status changes recorded in the logs of a specific protocol within a massive amount of text; or identifying the ports recorded in the logs of port oscillations caused by a transmission problem. After structuring the logs, they can be output in report formats (e.g., generating Excel format), making it much easier to statistically analyze various log variables. Therefore, there is often a need for log structuring in log analysis and processing.

[0004] Currently, the common method for extracting structured information from logs involves configuring a Database Description Language (DDL) file on the upstream system. This DDL file defines the parsing rules and field definitions for structured information. The upstream system then provides the logs and the log DDL file to the downstream system, allowing the downstream system to automatically extract structured data from the logs and subsequently load it into the target database for analysis.

[0005] In this method, the DDL file is pre-configured and fixed. However, in practical applications, different products or versions correspond to different logs, which means the DDL file needs to be modified according to changes in product and version, making maintenance difficult during the extraction of structured log information.

[0006] Application content

[0007] This application provides a data processing method and a data processing device, used to determine a log template (i.e., parsing rule) for the log type corresponding to multiple logs of the same type; then, based on the log template, variables of the multiple logs are extracted to generate structured logs. This eliminates the need for manual setting of parsing rules, thus avoiding manual maintenance and updates of the parsing rules during operation.

[0008] In a first aspect, embodiments of this application provide a data processing method, including:

[0009] After acquiring the log set, the data processing device determines that N logs in the log set belong to a type as the first type, where N is a positive integer. Then, the data processing device determines the log template corresponding to the first type based on the N logs. The log template corresponding to the first type is used to indicate the variable position of the N logs, that is, the parsing rule of the N logs. Finally, the data processing device extracts the variables of one or more logs from the N logs based on the variable position to generate a structured log file.

[0010] In this embodiment of the application, the log template corresponding to the first type can indicate not only the location of the variable, but also the number of variables, and other information corresponding to the log template corresponding to the first type, including but not limited to module name, severity, process ID, etc.

[0011] In this embodiment, the data processing device determines a log template (i.e., a parsing rule) for the log type corresponding to multiple logs of the same type; then, it extracts variables from the multiple logs based on the log template to generate a structured log file. That is, the data processing device can instantly acquire and update the parsing rules corresponding to the logs during operation, eliminating the need for manual setting of parsing rules and removing the need for manual maintenance and updates during operation.

[0012] Optionally, when determining the log template corresponding to the first type based on the N logs, the data processing device may use the following method:

[0013] The data processing device acquires the Mth log from the N logs, where M is a positive integer;

[0014] If M equals 1, the data processing device can use the Mth log as the log template corresponding to the first type; or, the data processing device can update the first target template determined by other logs of the same type as the N logs based on the Mth log as the log template corresponding to the first type.

[0015] If M is greater than or equal to 2, the data processing device can update the second target template determined by the M-1th log as the log template corresponding to the first type based on the Mth log.

[0016] The specific procedure by which the data processing device updates the first target template determined by other logs of the same type as the Nth log based on the Mth log is as follows:

[0017] The data processing device compares the Mth log entry with the first target template;

[0018] If the data processing device determines that there is a variable in the first target template relative to the Mth log, the data processing device uses a wildcard to represent the variable in the first target template relative to the Mth log as the log template corresponding to the first type. The wildcard is a pre-set character or string.

[0019] If the data processing device determines that the first target template does not have a variable relative to the Mth log entry, then the data processing device uses the first target template as the log template corresponding to the first type.

[0020] The specific steps by which the data processing device updates the second target template determined by the M-1th log entry as the log template corresponding to the first type based on the Mth log entry are as follows:

[0021] The data processing device compares the Mth log entry with the second target template;

[0022] If the data processing device determines that there is a variable in the second target template relative to the Mth log, the data processing device uses a wildcard to represent the variable of the second target template relative to the Mth log as the log template corresponding to the first type. The wildcard is a pre-set character or string, and the variable is the part of the second target template that is different from the Mth log.

[0023] If the data processing device determines that the second target template does not have a variable relative to the Mth log entry, then the data processing device uses the second target template as the log template corresponding to the first type.

[0024] In the technical solution provided in this application embodiment, the data processing device determines the final log template by comparing and updating the N logs in a cyclical manner. This ensures that logs in the same batch use the same log template, and when a new type of log appears, a new log template also appears. The log template can be obtained and updated in real time, thereby improving the accuracy of log data analysis.

[0025] Optionally, when the data processing device extracts variables from one or more of the N logs based on the variable positions in the log template corresponding to the first type to generate structured logs, it can use the following method:

[0026] In one possible implementation, the data processing device compares one or more logs from the N logs one by one with the log template corresponding to the first type, thereby determining the differences between one or more logs from the N logs and the log template corresponding to the first type, and identifying the differences as variables; then the data processing device extracts the variables to generate the structured log.

[0027] In another possible implementation, the data processing device obtains the variable position from the log template corresponding to the first type; then, the data processing device extracts the variable corresponding to the variable position from one or more logs among the N logs one by one to generate the structured log.

[0028] In the technical solution provided in this application embodiment, the data processing device can extract variables and generate structured logs in various ways according to the log template corresponding to the first type, which means that the logs can be processed flexibly and quickly.

[0029] Optionally, the data processing device can determine that the N logs in the log set belong to the first type based on a classification algorithm or a clustering algorithm. In practical applications, the classification algorithms that the data processing device can use include, but are not limited to, decision tree classification, Bayesian classification, BP neural network, and K-Means algorithms; clustering algorithms include, but are not limited to, SOM clustering and FCM clustering. There are also other algorithms, such as classifying logs by measuring the distance or relevance between them.

[0030] In the technical solution provided in this application embodiment, the data processing device can flexibly and quickly classify logs, effectively accelerating the log processing speed.

[0031] Optionally, the data processing device establishes a mapping relationship between the log template corresponding to the first type and the N logs through the index, that is, one template can correspond to multiple logs. Based on this, when structuring the N logs, the data processing device can query the log template corresponding to the first type of the N logs through the mapping relationship and the index. Then, the data processing device extracts the variables in the N logs according to the variable position in the log template corresponding to the first type to generate the structured log.

[0032] In the technical solution provided in this application embodiment, after the data processing device establishes the index mapping relationship between the log template corresponding to the first type and the N logs, it can effectively speed up the generation of the structured log by the data processing device and improve the log processing efficiency.

[0033] Optionally, after generating the structured log, the data processing device can also send the structured log along with the log template corresponding to the first type to the downstream system.

[0034] In the technical solution provided in this application embodiment, the data processing device sends the structured log and the log template corresponding to the structured log to the downstream system, enabling the downstream system to correctly analyze the structured log.

[0035] Optionally, the structured logs generated by the data processing device may also include, but are not limited to, any one or more of the following: time, hostname, template name, severity, and process identifier ID.

[0036] In the technical solution provided in this application embodiment, the more information the structured log contains, the more accurate the results will be when the downstream system analyzes the structured log.

[0037] Secondly, embodiments of this application provide a data processing device that performs the functions of the data processing device described in the above-described method. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions.

[0038] In one possible implementation, the data processing device includes:

[0039] The acquisition module is used to acquire a collection of logs.

[0040] The processing module is used to determine that N logs in the log set belong to the first type, where N is a positive integer; determine the log template corresponding to the first type based on the N logs, where the log template corresponding to the first type is used to indicate the variable position of the N logs; and extract variables from one or more logs of the N logs based on the variable position to generate structured logs.

[0041] In another possible implementation, the data processing device includes:

[0042] Transceiver, processor, and bus;

[0043] The transceiver is connected to the processor via the bus;

[0044] The transceiver performs the following steps:

[0045] Get the log collection;

[0046] The processor performs the following steps:

[0047] Determine that N logs in the log set belong to the first type, where N is greater than or equal to 1;

[0048] Based on the N log entries, determine the log template corresponding to the first type. The log template corresponding to the first type is used to indicate the variable position of the N log entries.

[0049] The variable position is extracted from one or more logs out of the N logs to generate a structured log.

[0050] Thirdly, embodiments of this application provide a computer storage medium storing program code for instructing the execution of the method described in the first aspect.

[0051] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: the data processing device determines the log template, i.e., the parsing rule, corresponding to the log type of multiple logs of the same type; then, it extracts the variables of the multiple logs according to the log template to generate structured logs. That is, there is no need to manually set the parsing rules, eliminating the need for manual maintenance and updating of the parsing rules during operation. Attached Figure Description

[0052] Figure 1 This is a system framework diagram of the log structure in the embodiments of this application;

[0053] Figure 2 This is a schematic diagram of one embodiment of the data processing method in this application;

[0054] Figure 3 This is a schematic diagram of one embodiment of the data processing device described in this application;

[0055] Figure 4 This is a schematic diagram of another embodiment of the data processing device in this application. Detailed Implementation

[0056] This application provides a data processing method and device for determining a log template (i.e., parsing rule) for the log type corresponding to multiple logs of the same type; then, it extracts variables from the multiple logs based on the log template to generate structured logs. This eliminates the need for manual setting of parsing rules, thus avoiding manual maintenance and updates of the parsing rules during operation.

[0057] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] Log data is one of the most important data types in system operation and maintenance. Log data analysis is essential for website user behavior analysis and system operation and maintenance statistics. However, directly analyzing a single log type variable from massive amounts of text logs is extremely difficult. Please refer to... Figure 1 As shown, currently, logs are typically structured and then output in a report format, making it easier to statistically analyze various log variables. Therefore, log structuring is a frequent requirement in log analysis and processing. Current methods for extracting structured log information usually involve configuring a DDL file in the upstream system, defining the parsing rules and field definitions for structured information. The upstream system then provides logs and the log DDL file to the downstream system, allowing the downstream system to automatically extract structured data from the logs and load it into the target database for subsequent analysis. In this method, the DDL file is pre-configured and fixed. However, in practical applications, different products or versions correspond to different logs, requiring the DDL file to be modified according to product or version changes, leading to maintenance difficulties during the extraction of structured log information.

[0059] To address this issue, the following solution is provided in this application embodiment: After acquiring the log set, the data processing device determines that N logs in the log set belong to a type as a first type, where N is a positive integer; then, the data processing device determines the log template corresponding to the first type based on the N logs, and the log template corresponding to the first type is used to indicate the variable position of the N logs, i.e., the parsing rule of the N logs; finally, the data processing device extracts the variables of one or more logs from the N logs based on the variable position to generate a structured log file.

[0060] Please refer to the details. Figure 2 As shown, one embodiment of the data processing method in this application includes:

[0061] 201. Data processing equipment acquires log sets.

[0062] The data processing device acquires a collection of logs.

[0063] In practical applications, the data processing device can acquire the first log file in various ways, including but not limited to importing via an interface and transmitting via an interface; the specific methods are not limited here. The log set includes, but is not limited to, system logs (Syslog). The data processing device typically acquires a large number of logs, and therefore usually processes them in batches during the log structuring process. That is, the data processing device will only classify the acquired logs into a log set when the log acquisition time meets a preset duration or the number of logs acquired meets a preset threshold. Then, it will classify each log in the log set according to its type and determine the template corresponding to each log type, i.e., the parsing rules. The preset duration and preset threshold are predetermined, and their specific values ​​are not limited here.

[0064] 202. The data processing device determines that N logs in the log set belong to the first type.

[0065] The data processing device parses each log in the log set and groups N logs of the same type into one category, namely the first type.

[0066] In practical applications, this data processing device can determine the type of each log in the log set in various ways. For example, it can determine the type of the first log file using classification or clustering algorithms, or it can determine the type of the first log file by obtaining its source code. The classification algorithms that the data processing device can use include, but are not limited to, decision tree classification, Bayesian classification, backpropagation (BP) neural network, and K-Means algorithms; the clustering algorithms include, but are not limited to, SOM clustering and FCM clustering. There are also other algorithms, such as classifying logs by measuring the distance or relevance between them.

[0067] 203. The data processing device determines the log template corresponding to the first type based on the N logs. The log template corresponding to the first type is used to indicate the variable position of the N logs.

[0068] The data processing device determines the log template corresponding to the first type based on the N logs. The log template corresponding to the first type is used to indicate the variable position of the N logs.

[0069] In practical applications, when the data processing device determines the log template corresponding to the first type based on the N logs, it can do so in the following way:

[0070] The data processing device acquires the Mth log from the N logs, where M is a positive integer;

[0071] If M equals 1, the data processing device can use the Mth log as the log template; or, the data processing device updates the first target template determined by other logs of the same type as the N logs based on the Mth log as the log template. If M is greater than or equal to 2, the data processing device can update the second target template determined by the (M-1)th log as the log template based on the Mth log as the log template. Specifically, the data processing device updates the first target template determined by other logs of the same type as the N logs based on the Mth log as the log template as follows: The data processing device compares the Mth log with the first target template; if the data processing device determines that the first target template has a variable relative to the Mth log, the data processing device uses a wildcard to represent the variable of the first target template relative to the Mth log as the log template, where the wildcard is a pre-set character or string; if the data processing device determines that the first target template does not have a variable relative to the Mth log, the data processing device uses the first target template as the log template. The data processing device updates the second target template determined by the (M-1)th log entry based on the Mth log entry as the log template. The specific steps are as follows: The data processing device compares the Mth log entry with the second target template. If the data processing device determines that the second target template has a variable relative to the Mth log entry, then the data processing device uses a wildcard to represent the variable of the second target template relative to the Mth log entry as the log template. The wildcard is a pre-set character or string. If the data processing device determines that the second target template does not have a variable relative to the Mth log entry, then the data processing device uses the second target template as the log template. The wildcard can be a character or a string, such as an asterisk "*", an exclamation mark "!", a hash symbol "#", or multiple asterisks "***". In this embodiment, an asterisk "*" is used as an example.

[0072] For example, the log set includes four logs, as shown in Table 1. According to the algorithm, the logs shown in the second row and the third row are of the same type, and the logs shown in the fourth row and the fifth row are of the same type. In this embodiment, the logs shown in the second row and the third row are used as examples.

[0073]

[0074]

[0075] Table 1

[0076] The specific method by which the data processing device determines the log template corresponding to the type of the two logs is as follows: When the data processing device obtains the first log from the two logs, that is, the log shown in the second row of Table 1, the data processing device can use the first log as the log template, that is, the log template at this time is as shown in Table 2:

[0077] template The BFD session went Down.SessName is 26585-tdm,Interface is GE1 / 1 / 0

[0078] Table 2

[0079] When the data processing device retrieves the second log from the first two logs (the log shown in the third row of Table 1), it needs to update the log template determined by the first log based on this second log (the log template shown in Table 2). The device then compares the second log with the log template shown in Table 2, using a wildcard * to represent the differences between the log template shown in Table 2 and the second log, thereby generating the log template shown in Table 3. In this case, the log template of the first type corresponding to the two logs is the log template shown in Table 3.

[0080] template The BFD session went Down.SessName is*,Interface is*

[0081] Table 3

[0082] In practical applications, when the data processing device obtains the first log entry, it can generate the log template in the following manner: The data processing device obtains a first target template determined in the previous batch of log processing based on logs of the same type as the first type in the embodiments of this application. The first target template is shown in Table 3. The data processing device can then compare the first log entry with the first target template to determine whether there are variables in the first target template relative to the first log entry. If there are variables, the data processing device uses the wildcard * to represent the words in the first target template that are variables relative to the first log entry to generate the log template, as shown in Table 4. If there are no variables, the data processing device uses the first target template, i.e., Table 3, as the log template. Then, when the data processing device obtains the second log entry, it compares it with the log template shown in Table 4. If a variable exists, the data processing device uses the wildcard * to represent the word in the first target template that is a variable relative to the first log entry, as shown in Table 4. If no variable exists, the data processing device uses the first target template, i.e., Table 3, as the log template. In practical applications, if the first log entry is "The BFDsession went Up.SessName is 26585-tdm,Interface is GE1 / 1 / 0", the data processing device updates the first target template shown in Table 3 based on the first log entry. The resulting log template of this first type is shown in Table 5.

[0083] template The BFD session went Down.SessName is*,Interface is*

[0084] Table 4

[0085] template The BFD session went*.SessName is*,Interface is*

[0086] Table 5

[0087] In practical applications, the data processing device can also establish a mapping relationship between the N logs and the log template corresponding to the first type. Thus, when the data processing device generates a structured log from one or more of the N logs based on the log template corresponding to the first type, it can quickly query the logs from the N logs according to the mapping relationship.

[0088] In this embodiment of the application, when the data processing device determines the log template corresponding to the first type determined by the N logs, one possible implementation is that the data processing device can directly traverse the log set and generate log templates for all logs simultaneously according to the type label. For example, if the log set contains five logs, where logs 1, 3, and 4 are logs of the first type, and logs 2 and 5 are logs of the second type, then when the data processing device traverses the log set, the following situations may occur: the first log obtained by the data processing device is log 1, and the data processing device uses log 1 as the log template of the first type; then the second log obtained by the data processing device is log 2, and the data processing device knows from the type label that the second log is of a different type from the first log, so the data processing device uses log 2 as the first log of the second type, and the data processing device uses log 2 as the first log of the second type. As a second type of log template; next, the data processing device obtains a third log, log 3. Based on the type label, the data processing device knows that the third log is of the same type as the first log. Therefore, the data processing device updates the log template determined by the first log based on the third log, and the resulting log template is used as the first type of log template. The data processing device obtains a fourth log, log 4. Based on the type label, the data processing device knows that the fourth log is of the same type as both the third and first logs. Therefore, the data processing device updates the log template determined by the third log based on the fourth log, and the final log template is used as the first type of log template. Finally, the data processing device obtains a fifth log, log 5. Based on the type label, the data processing device knows that the fifth log is of the same type as the second log. Therefore, the data processing device updates the log template determined by the first log based on the fifth log, and the final log template is used as the second type of log template. Of course, the examples in this embodiment are only one possible scenario, and specific scenarios are not limited here. In another possible implementation, the data processing device can iterate through the logs of each type to obtain the corresponding log templates for each type of log. For example, if the log set is shown in Table 1, the data processing device may iterate through different log types as follows: the data processing device first iterates through the logs shown in the second and third rows of Table 1, and then iterates through the logs shown in the fourth and fifth rows of Table 1. The logs shown in the second and third rows of Table 1 are of the first type, and the logs shown in the fourth and fifth rows of Table 1 are of the second type.The specific steps are as follows: The data processing device acquires the first log of the first type, i.e., the log shown in the second row of Table 1, and uses this first log as the log template for the first type. Then, the data processing device acquires the second log of the first type, i.e., the log shown in the third row of Table 1, and updates the log template determined by the first log based on the second log to generate the final log template for the first type. Next, the data processing device acquires the first log of the second type, i.e., the log shown in the fourth row of Table 1, and uses this first log of the second type as the log template for the second type. Finally, the data processing device acquires the second log of the second type, i.e., the log shown in the fifth row of Table 1, and updates the log template determined by the first log of the second type based on the second log to generate the final log template for the second type. Of course, the examples in this embodiment are only one possible scenario, and the specific scenario is not limited here.

[0089] In this embodiment of the application, when the data processing device determines the log template corresponding to the first type based on the N logs, the data processing device can also separate the N logs using statement delimiters to generate word vectors. The statement delimiters here are preset characters or preset strings, such as an asterisk "*", a space "", a comma "," etc., and the specific selection is not limited here.

[0090] 204. The data processing device extracts the variable from one or more logs out of the N logs based on the variable's position to generate a structured log.

[0091] The data processing device extracts the variables from the N log entries based on the variable's location to generate a structured log.

[0092] In practical applications, this data processing device can structure the N log entries in the following way:

[0093] In one possible implementation, the data processing device directly compares each of the N logs with the log template. Then, the data processing device identifies the differences between each of the N logs and the log template as variables and extracts these variables to generate the structured log. For example, in this embodiment, the data processing device compares the log template shown in Table 3 with the two logs in Table 1 to determine the differences between the two logs and the log template. The data processing device then extracts these differences to generate the structured log as shown in Table 6.

[0094] In another possible implementation, the data processing device parses the log template to obtain information about the variable positions within it. Then, the data processing device extracts the variables corresponding to those positions from each of the N log entries to generate the structured log. For example, in the template shown in Table 3 of this application, if the data processing device treats the punctuation mark and the word as a unit, it can determine that the variable positions in the log template are the 8th and 11th word positions. Therefore, when traversing the two log entries shown in Table 1, the data processing device can directly extract the variables from the 8th and 11th word positions of these two log entries to generate the structured log as shown in Table 6.

[0095] Variable 1 Variable 2 26585-tdm GE1 / 1 / 0 26586-tdm GE1 / 2 / 0

[0096] Table 6

[0097] In practical applications, this structured log can also include information such as time, hostname, and module. Taking the structured log shown in Table 6 as an example, the structured log is shown in Table 7:

[0098]

[0099] Table 7

[0100] After generating structured logs, the data processing device can send the structured logs along with the log template to downstream systems, allowing the downstream systems to parse the structured logs based on the log template.

[0101] In this embodiment, the data processing device determines a log template (i.e., a parsing rule) for the log type corresponding to multiple logs of the same type; then, it extracts variables from the multiple logs based on the log template to generate a structured log. That is, the data processing device can instantly acquire and update the parsing rules corresponding to the logs during operation, eliminating the need for manual setting of parsing rules and avoiding manual maintenance and updates during operation. The data processing method in this embodiment has been described above; the data processing device in this embodiment is described below.

[0102] Please refer to the details. Figure 3 As shown in the embodiments of this application, the data processing device includes:

[0103] Module 301 is used to retrieve the log collection;

[0104] Processing module 302 is used to determine that N logs in the log set belong to a first type, where N is a positive integer; determine a log template corresponding to the first type based on the N logs, where the log template corresponding to the first type is used to indicate the variable position of the N logs; and extract variables from one or more logs among the N logs based on the variable position to generate structured logs.

[0105] Optionally, the processing module 302 is specifically used to obtain the Mth log from the N logs, where M is a positive integer;

[0106] When M equals 1, the Mth log entry is used as the log template corresponding to the first type.

[0107] or,

[0108] The first target template is updated based on the Mth log entry as the log template corresponding to the first type. The first target template is the log template determined by other logs of the same type as the Nth log entry.

[0109] Optionally, when M is greater than or equal to 2, the processing module 302 is specifically used to update the second target template as the log template corresponding to the first type according to the Mth log, wherein the second target template is the log template determined by the data processing device according to the (M-1)th log.

[0110] Optionally, the processing module 302 is specifically used to compare the Mth log entry with the second target template;

[0111] If it is determined that there is a variable in the second target template relative to the Mth log entry, then the variable in the second target template relative to the Mth log entry is represented by a wildcard and used as the log template corresponding to the first type. The wildcard is a pre-set character or string.

[0112] If it is determined that the second target template does not have a variable relative to the Mth log entry, then the second target template is used as the log template corresponding to the first type.

[0113] Optionally, the processing module 302 is specifically used to identify the differences between one or more logs obtained through comparison and the log template corresponding to the first type as variables; and extract the variables to generate the structured log.

[0114] Optionally, the processing module 302 is specifically used to: obtain the position of the variable recorded in the log template corresponding to the first type; and extract the variable corresponding to the position of the variable from one or more logs in the N logs to generate the structured log.

[0115] Optionally, the processing module 302 is specifically used to determine, based on a classification algorithm or a clustering algorithm, that the N logs in the log set belong to the first type.

[0116] Optionally, the processing module 302 is further configured to establish a mapping relationship between the log template corresponding to the first type and the N logs;

[0117] The processing module 302 is specifically used to query one or more logs from the N logs corresponding to the log template of the first type according to the mapping relationship;

[0118] The structured log is generated by extracting the variable from the N logs based on the position of the variable in the log template corresponding to the first type.

[0119] Optionally, the data processing device further includes a sending module 303 for sending the structured log and the log template corresponding to the first type to the downstream system.

[0120] Optionally, the structured log may also include one or more of the following: time, hostname, module name, severity, and process ID.

[0121] Furthermore, Figure 3 The data processing equipment in the middle can also be used to perform Figure 2 Any step performed by the data processing device in the process, to achieve Figure 2 Any function that the data processing equipment in the middle can perform.

[0122] In this embodiment, the processing module 302 determines a log template (i.e., a parsing rule) for the log type corresponding to multiple logs of the same type. Then, the processing module 302 extracts variables from the multiple logs based on the log template to generate structured logs. This means that the data processing device can instantly acquire and update the parsing rules corresponding to the logs during operation, eliminating the need for manual setting of parsing rules and removing the need for manual maintenance and updates during operation.

[0123] Please refer to the details. Figure 4 As shown, in another embodiment of the data processing device in this application, the data processing device includes:

[0124] Transceiver 401, processor 402, bus 403; the transceiver 401 and the processor 402 are connected via the bus 403.

[0125] Bus 403 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0126] Processor 402 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0127] The processor 402 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0128] See Figure 4 As shown, the data processing device may further include a memory 404; the memory 404 may also be used to store a log set; the memory 404 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 404 may also include a combination of the above types of memory.

[0129] Optionally, memory 404 can also be used to store program instructions. Processor 402 can execute the program instructions stored in memory 404 by calling them. Figure 2 One or more steps in the illustrated embodiments, or optional implementations thereof, achieve the functionality of the data processing device behavior in the above method. In this embodiment, the transceiver 401 executes... Figure 2 Step 201 is shown;

[0130] The processor executes Figure 2 Steps 202 to 204 are shown.

[0131] In this embodiment, the processor 402 determines a log template (i.e., a parsing rule) for the log type corresponding to multiple logs of the same type. Then, the processor 402 extracts variables from the multiple logs based on the log template to generate a structured log. This means that the data processing device can instantly acquire and update the parsing rules corresponding to the logs during operation, eliminating the need for manual setting of parsing rules and removing the need for manual maintenance and updates during operation.

[0132] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate; the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of this embodiment. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: The data processing device acquires a log set, the log set including N logs, where N is a positive integer, and the N logs belong to a first type; The data processing device compares the first log in the N logs with the second log in the N logs to determine a log template including a first wildcard, the first wildcard being used to indicate the variable position between the first log and the second log, and the log template being the log template corresponding to the first type; The data processing device compares the log template with the target log in the log set to extract the difference in the target log. The difference includes the variable content corresponding to the first wildcard, and the target log belongs to the first type. The data processing device generates a structured log based on the differences.

2. The method according to claim 1, characterized in that, The method further includes: Based on the third log in the N logs, update the log template. The updated log template includes a second wildcard, which is used to indicate the variable position between the log template before the update and the third log. The step of comparing the log template and the target log in the log set to extract the differences in the target log includes: The updated log template and the target log in the log set are compared to extract the differences in the target log. The differences include the variable contents corresponding to the first wildcard and the second wildcard.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The data processing device determines that the N log entries belong to the first type; The data processing device determines that the target log belongs to the first type.

4. The method according to claim 3, characterized in that, The data processing device determines that the N log entries belong to the first type, including: The data processing device determines that the N logs belong to the first type based on a classification algorithm or a clustering algorithm.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The data processing device sends the structured log and the log template to downstream devices.

6. The method according to any one of claims 1 to 5, characterized in that, The structured log includes any one or more of the following: time, hostname, module name, severity, and process identifier ID.

7. A data processing device, characterized in that, include: The acquisition module is used to acquire a log set, the log set including N logs, where N is a positive integer, and the N logs belong to the first type; The processing module is configured to compare a first log in the N logs with a second log in the N logs to determine a log template including a first wildcard, wherein the first wildcard is used to indicate the variable position between the first log and the second log, and the log template is a log template corresponding to the first type; and to compare the log template with a target log in the log set to extract the difference portion in the target log, wherein the difference portion includes the variable content corresponding to the first wildcard, and the target log belongs to the first type; Structured logs are generated based on the differences.

8. The device according to claim 7, characterized in that, The processing module is further configured to: update the log template based on the third log among the N logs, wherein the updated log template includes a second wildcard, the second wildcard being used to indicate the variable position between the log template before the update and the third log; compare the updated log template with the target log in the log set to extract the difference portion in the target log, the difference portion including the variable content corresponding to the first wildcard and the second wildcard.

9. The device according to claim 7 or 8, characterized in that, The device also includes: The determination module is used to determine that the N logs belong to the first type; and to determine that the target log belongs to the first type.

10. The device according to claim 9, characterized in that, The determining module is also used to determine, based on a classification algorithm or a clustering algorithm, that the N log entries belong to the first type.

11. The device according to any one of claims 7 to 10, characterized in that, The device also includes: The sending module is used to send the structured log and the log template to downstream devices.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-6.

13. A chip, characterized in that, The chip is used to perform the method as described in any one of claims 1-6.

Images