A distributed log collection method, computer equipment and storage medium

Abstract

The present invention provides a distributed log collection method, computer equipment and storage medium, wherein the method comprises: receiving a new configuration template sent by a server, and determining whether the new configuration template is the same as an old configuration template in a collection rule base; if not, adding the new configuration template to the collection rule base; if so, replacing the old configuration template with the new configuration template; collecting corresponding log files according to all configuration templates in the collection rule base; uploading the collected log files to the server. The method solves the problem that log collection in active mode is cumbersome and inefficient and time-effective.

Description

Technical Field

[0001] The present invention relates to the technical field of log collection, and in particular to a distributed log collection method, a computer device and a storage medium. Background Art

[0002] In the process of using devices and services, a large amount of log data will be generated. Operation and maintenance personnel need to collect the logs of all these devices and effectively extract key information from the massive logs to find relevant security events and causes of failures, such as user access, unusual activities, abnormal user behavior, policy violations, internal threats, external attacks, data theft, etc. for maintenance.

[0003] Traditional log collection systems all adopt active mode. First, the collection program distributed on each node collects relevant logs and data according to the configuration file saved locally. After information extraction, formatting, data filtering and other processing, it is sent to the remote server. The server receives the log data and saves it for users to query and operate. However, this mode has the following disadvantages:

[0004] 1) The collection program on the node collects and processes log data according to the configuration file, which is not conducive to the maintenance of large-scale equipment. When the device log is updated, the log collection path is changed, the log data processing rule is changed, the log server is changed, and new equipment is put on the shelves, the configuration files on the massive nodes need to be updated, which is cumbersome and prone to errors, and the timeliness is not ideal;

[0005] 2) The server needs to be equipped with high-performance hardware to filter and format massive amounts of log data, which is very costly. Summary of the invention

[0006] The present invention aims to overcome at least one defect in the prior art and provide a distributed log collection method, a computer device and a storage medium to solve the problem that log collection in active mode is cumbersome and has low timeliness.

[0007] In a first aspect, a distributed log collection method is provided, comprising:

[0008] Receive a new configuration template sent by the server, and determine whether the new configuration template is the same as the old configuration template in the collection rule base;

[0009] If not, then adding the new configuration template to the collection rule base;

[0010] If yes, replacing the old configuration template with the new configuration template;

[0011] According to the log files corresponding to all the configuration templates in the collection rule library, the collected log files are uploaded to the server.

[0012] After receiving the new configuration template sent by the server, the new configuration template is added to the collection rule library or the old configuration template in the collection rule library that is the same as the new configuration template is replaced, and the corresponding log files are collected according to all the configuration templates in the collection rule library. This passive mode log collection method can quickly collect various types of terminal node device log files that are large in number, widely distributed, and have non-uniform log formats, and can quickly adjust the log file collection content according to business conditions and operation and maintenance requirements.

[0013] Further, the content of the configuration template includes a collection period and a collection rule; collecting corresponding log files according to all configuration templates in the collection rule library, and uploading the collected log files to the server, including:

[0014] Reading a collection period of a configuration template, and reading a collection rule of the configuration template once in each collection period;

[0015] Reading log files that have not been uploaded to the server;

[0016] Selecting one of the log files as the current log file, grouping the log files according to the read grouping rule, obtaining a grouping result of the current log file, and uploading the grouping result of the current log file to the server;

[0017] It is determined whether all the read log files have been uploaded to the server. If not, the next log file is continuously collected according to the read collection rule until all the read log files have been uploaded to the server.

[0018] Furthermore, the aggregation rule includes a splitting rule and a plurality of matching rules arranged in sequence;

[0019] The log file is currently aggregated according to the read aggregation rule, and the aggregation result of the current log file is obtained, including:

[0020] Splitting the current log file into log data according to the splitting rule;

[0021] Selecting one of the log data as the current log data, and selecting one of the matching rules as the current matching rule according to the arrangement order of the matching rules;

[0022] Match the current log data using the current matching rule, and determine whether the configuration of the current matching rule indicates to continue matching the current log data;

[0023] If it is indicated to continue matching the current log data, determining whether there is a matching rule that does not match the current log data;

[0024] If there is a matching rule that does not match the current log data, then according to the order of the matching rules, the next matching rule is selected as the current matching rule, and the current matching rule is continued to be used to match the current log data until there is no matching rule that does not match the current log data, and the matching result of the current log data is obtained;

[0025] If the instruction is not to continue matching the current log data, the matching of the current log data is terminated, and it is determined whether there is the next log data that does not match the matching rule;

[0026] If yes, the next log data is selected as the current log data, and a matching rule is selected as the current matching rule according to the order of matching rules, and the current matching rule is used to match the current log data:

[0027] If not, the collection result of the current log file is obtained according to the matching results of all log data.

[0028] Furthermore, the configuration of the current matching rule includes a matching success configuration and a matching failure configuration; the current log data is matched using the current matching rule, including:

[0029] Match the current log data according to the regular expression in the current matching rule, and determine whether the regular expression is matched successfully;

[0030] If yes, configure the current log data according to the successful matching configuration of the current matching rule to obtain the matching sub-result of the current matching rule;

[0031] If not, configure the current log data according to the match failure configuration of the current matching rule;

[0032] The matching result of the current log data is composed of the matching sub-results of all matching rules in the order of arrangement.

[0033] Furthermore, before matching the current log data according to the regular expression in the current matching rule, the method further includes:

[0034] Determine whether the regular expression in the current matching rule is empty;

[0035] If not, matching the current log data according to the regular expression;

[0036] If so, the current log data is configured according to the matching failure configuration of the current matching rule, and the next matching rule is selected as the current matching rule according to the arrangement order of the matching rules, and the current log data is continued to be matched using the current matching rule.

[0037] Furthermore, the matching sub-result and the matching result are messages in json format;

[0038] Configure the current log data according to the successful matching configuration of the current matching rule, including:

[0039] Match the current log data according to the regular expression, and obtain the log character string in the current log data that matches the regular expression;

[0040] According to the successful matching configuration of the current matching rule, the matching of the current log data is ended, or a flag string is inserted into the tag of the message in the json format to mark the log string, and the log string with the flag string is obtained or discarded to obtain the matching sub-result of the current matching rule;

[0041] Configure the current log data according to the match failure configuration of the current matching rule, including:

[0042] According to the match failure configuration of the current matching rule, the matching of the current log data is ended, or a default value is inserted into the tag of the message in the json format.

[0043] Log data is matched according to regular expressions, and the log data is configured according to the match failure configuration or the match success configuration of the matching rule to obtain the matching sub-result in json format, and then the matching sub-results of all matching rules are composed of the matching result in json format, so that the log file is formatted before being uploaded to the server, which solves the problem of insufficient server processing capacity due to processing large batches of log formats.

[0044] Furthermore, the content of the configuration template also includes a log path; reading a log file that has not been uploaded to the server includes:

[0045] According to the log path, the log files that have not been uploaded to the server are read.

[0046] Furthermore, the aggregation rule also includes a separation mark;

[0047] Before selecting the next log data as the current log data, the method further includes: setting the separation mark after the matching result of the current log data to separate the matching result of the current log data from the matching result of the next log data.

[0048] In a second aspect, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0049] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the method described above when executed by a processor.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows: after receiving the new configuration template sent by the server, the present invention adds the new configuration template to the collection rule library or replaces the old configuration template in the collection rule library that is the same as the new configuration template, and collects the corresponding log files according to all the configuration templates in the collection rule library. Through this passive mode log collection method, various types of terminal node device log files with a large number, wide distribution, and non-uniform log formats can be quickly collected, and the log file collection content can be quickly adjusted according to business conditions and operation and maintenance requirements; the log file formatting can be completed according to the collection rules at the terminal node where the log file is generated, which solves the problem of insufficient server processing capacity due to processing large batches of log formats; and the lost log retransmission function can also be quickly and easily implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the process described in Example 1;

[0052] Figure 2 is another schematic flow chart of the method described in Example 1;

[0053] Figure 3 is another schematic flow chart of the method described in Example 1;

[0054] Figure 4 Another schematic diagram of the method described in Example 1. DETAILED DESCRIPTION

[0055] The drawings of the present invention are only for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0056] Example 1

[0057] In this embodiment, if Figure 1 As shown, a distributed log collection method is provided, with the terminal node as the execution subject, and the method includes the following steps:

[0058] S100. Receive a new configuration template sent by a server, and determine whether the new configuration template is the same as an old configuration template in a collection rule base.

[0059] The configuration template is set by the administrator on the server to control the terminal node to collect log files according to the content of the configuration template. Before issuing a new configuration template, the server first reads all configuration templates from the configuration file and filters the enabled configuration templates; reads the content of the enabled configuration template and compares the read content with the content of the saved configuration template to determine whether it is a new configuration template or whether the content of the configuration template has been updated. If so, the configuration template is formatted to form a new configuration template, and is issued to the corresponding terminal node according to the new configuration template; the content of the configuration template includes but is not limited to the following fields:

[0060] The name of the configuration template Name, whether the configuration template is enabled Active, the server address Host that receives the collected log files, the address Node of the terminal node that uses the configuration template to collect log files, the log path Path of the log files to be collected, the collection cycle Time of the log files, the separation mark Split, whether to resend all log files Resend, and the matching rule Method. Among them, Active: True means that the configuration template is enabled, and Active: False means that the configuration template is not enabled; the log path Path of the log files to be collected can be configured in array form, so that one configuration template can be used by multiple terminal nodes.

[0061] Preferably, the server reads all configuration templates from the configuration file, and filters the configuration templates with Active: True, compares the content of the read enabled configuration template with the content of the saved configuration template, and when the name Name of the configuration template is different from the name Name of the saved configuration template, it is determined that the configuration template is a newly added template, and when the instruction information configured in the remaining fields of the configuration template is not completely the same as the instruction information configured in the remaining fields saved in the configuration template, it is determined that the content of the configuration template is updated; the configuration template is formatted as a new configuration template, and the new configuration template is sent to the terminal node of the corresponding address according to the address Node of the terminal node that uses the configuration template for log file collection.

[0062] S200. If not, then add the new configuration template to the collection rule base.

[0063] After receiving the new configuration template sent by the server, the terminal node first determines whether the new configuration template is the same as the old configuration template in the collection rule base, and the collection rule base stores several configuration templates sent by the server. Preferably, the terminal node determines whether the name of the new configuration template is the same as the name of the old configuration template in the collection rule base. When it is determined that the name of the new configuration template is different from the names of all the old configuration templates in the collection rule base, the new configuration template is added to the collection rule base.

[0064] S300. If yes, replace the old configuration template with the new configuration template.

[0065] When the terminal node determines that the name of the new configuration template is the same as the name of an old configuration template in the collection rule, it determines whether the content of the new configuration template is the same as the content of the old configuration template with the same name. If not, the content of the new configuration template replaces the content of the old configuration template, that is, the content of the old configuration template is updated; the terminal node collects the corresponding log files according to all configuration templates in the collection rule library.

[0066] S400. Collect corresponding log files according to all configuration templates in the collection rule library, and upload the collected log files to the server.

[0067] Specifically, the content of the configuration template includes a collection period and a collection rule; Step S400 collects the corresponding log files according to all the configuration templates in the collection rule library, and uploads the collected log files to the server, such as Figure 2 As shown, the following steps may be specifically included:

[0068] S410. Read the collection period of the configuration template, and read the collection rule of the configuration template once in each collection period.

[0069] The terminal node reads each configuration template in the collection rule library, and reads the collection period Time of each configuration template, and reads the collection rule of the configuration template once in each collection period Time of each configuration template; for example, when the collection period Time of a configuration template is 5s, after reading the collection period of the configuration template, the terminal node reads the collection period of the configuration template every 5s.

[0070] S420. Read the log files that have not been uploaded to the server.

[0071] The content of the configuration template also includes a log path; step S420 of reading the log file that has not been uploaded to the server may specifically include:

[0072] According to the log path, the log files that have not been uploaded to the server are read.

[0073] The terminal node reads the log files that have not been uploaded to the server according to the log path Path of the log files to be collected. It can be understood that the log files that have not been uploaded to the server can be log files generated by the terminal node after a certain time point, or can be all log files generated by the terminal node. When Resend: True in the configuration template is read, it means retransmitting all log files under the log path, that is, all log files are log files that have not been uploaded to the server. When Resend: False in the configuration template is read, it can be judged according to the modification time of the log file, the location of the log file that has not been uploaded to the server read in the previous collection cycle, and the time of the previous collection cycle. For example, the time of the log file that has not been uploaded to the server read in the previous collection cycle is 10:19:00, then the log file whose modification time is after 10:19:00 is the log file that has not been uploaded to the server.

[0074] S430. Select one of the log files as the current log file, aggregate the current log file according to the read aggregation rule, obtain the aggregation result of the current log file, and upload the aggregation result of the current log file to the server.

[0075] The collection rule includes a splitting rule and a plurality of matching rules arranged in sequence; Step S403 collects the current log file according to the read collection rule, such as Figure 3 As shown, the following steps may be specifically included:

[0076] S431. Split the current log file into log data according to the splitting rule.

[0077] Specifically, the log file may be split according to the generation time of each log data in the log file, and the log file may be split into one or more log data.

[0078] S432. Select one of the log data as the current log data, and select one of the matching rules as the current matching rule according to the arrangement order of the matching rules;

[0079] S433. Use the current matching rule to match the current log data, and determine whether the configuration of the current matching rule indicates to continue matching the current log data.

[0080] Specifically, the order of the matching rule Method can be arranged according to the order of the Method id, and the configuration of the current matching rule includes a matching success configuration and a matching failure configuration; wherein, step S433 uses the current matching rule to match the current log data, such as Figure 4 As shown, it may specifically include:

[0081] Determine whether the regular expression in the current matching rule is empty;

[0082] If it is determined that the regular expression is not empty, the current log data is matched according to the regular expression, and it is determined whether the regular expression is matched successfully;

[0083] If the regular expression is determined to be matched successfully, the current log data is configured according to the matching success configuration of the current matching rule;

[0084] Among them, the current log data is configured according to the successful matching configuration of the current matching rule, including:

[0085] Match the current log data according to the regular expression, and obtain the log character string in the current log data that matches the regular expression;

[0086] According to the successful configuration of the match of the current matching rule, the matching of the current log data is ended, or a flag string is inserted into the tag of the message in the json format to mark the log string, the log string with the flag string is obtained or discarded, and the matching sub-result of the current matching rule is obtained;

[0087] If the regular expression is determined to be empty or the regular expression fails to match, the current log data is configured according to the match failure configuration of the current matching rule;

[0088] Among them, the current log data is configured according to the matching failure configuration of the current matching rule, including:

[0089] According to the match failure configuration of the current matching rule, the matching of the current log data is ended, or a default value is inserted into the tag of the message in the json format.

[0090] Specifically, the matching rule Method includes Method id, regular expression Method regex, matching success configuration Method success and matching failure configuration Method failed; the matching success configuration Method success can be insert-insert the flag string that meets the rule into the tag tag in the JSON format message, or ignore-discard the log string, or end-terminate the match; the matching failure configuration Method failed can be insert-insert default into the tag tag in the json format message, or end-terminate the match; taking one of the matching rules as an example, the matching rule is {"id":3,"regex":"\S+","success":"insert","failed":"insert","tag":"hostname","default":""}, then the regular expression of the matching rule is not empty, and the current log data is matched according to the regular expression regex: \S+. If the match is successful, the first log string that meets the rule is intercepted and saved, and the flag string hostname is inserted into the tag tag in the json format message, and the log string with the flag string hostname is obtained; if the match fails, the default value default is inserted into the tag tag in the json format message, and go to step S434.

[0091] The matching result of the current log data is composed of the matching sub-results of all matching rules in the order of arrangement.

[0092] S434. If the instruction is to continue matching the current log data, determine whether there are any matching rules that do not match the current log data.

[0093] Specifically, when the matching is successfully configured and Method success is not end, it indicates to continue matching the current log data.

[0094] If so, the next matching rule is selected as the current matching rule according to the order of the matching rules, and the current matching rule is continued to be used to match the current log data until there is no matching rule that does not match the current log data, thereby obtaining the matching result of the current log data.

[0095] S435. If it is indicated not to continue matching the current log data, the matching of the current log data is terminated, and it is determined whether there is the next log data that does not match the matching rule.

[0096] Specifically, when the match is successfully configured and Method success is set to end, it indicates that the matching of the current log data ends.

[0097] S436. If yes, then select the next log data as the current log data, continue to select a matching rule as the current matching rule according to the arrangement order of the matching rules, and use the current matching rule to match the current log data.

[0098] S437. If not, obtain the collection result of the current log file based on the matching results of all log data.

[0099] Specifically, after obtaining the matching result in json format, the matching result is uploaded to the corresponding server according to the server address Host that receives the collected log file.

[0100] S440. Determine whether all the read log files have been uploaded to the server. If not, continue to collect the next log file according to the read collection rule until all the read log files have been uploaded to the server.

[0101] After receiving the new configuration template sent by the server, the terminal node of this implementation adds the new configuration template to the collection rule library or replaces the old configuration template in the collection rule library that is the same as the new configuration template, and collects the corresponding log files according to all the configuration templates in the collection rule library. This passive mode log collection method can quickly collect various types of terminal node device log files that are large in number, widely distributed, and have non-uniform log formats, and can quickly adjust the log file collection content according to business conditions and operation and maintenance requirements.

[0102] In one embodiment, the aggregation rule further includes a separation mark;

[0103] Before selecting the next log data as the current log data, the method further includes: setting the separation mark after the matching result of the current log data to separate the matching result of the current log data from the matching result of the next log data.

[0104] Example 2

[0105] In this embodiment, a computer device is provided, which may be a terminal node, and includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities, the memory includes an operating system in a non-volatile storage medium and an operating environment for a computer program, and the communication interface is used to communicate with an external terminal node in a wired or wireless manner, and the wireless manner may be implemented via WiFi, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a distributed log collection method described in Example 1 is implemented.

[0106] Example 3

[0107] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the distributed log collection method described in Embodiment 1 is implemented.

[0108] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A distributed log collection method, characterized in that: include: Receive a new configuration template sent by the server, and determine whether the new configuration template is the same as the old configuration template in the collection rule base; If not, then adding the new configuration template to the collection rule base; If yes, replacing the old configuration template with the new configuration template; Collecting corresponding log files according to all configuration templates in the collection rule library, and uploading the collected log files to the server; The content of the configuration template includes the collection cycle and collection rules; According to the log files corresponding to all the configuration templates in the collection rule library, uploading the collected log files to the server includes: Reading the collection period of the configuration template, and reading the collection rule of the configuration template once in each collection period; reading the log file that has not been uploaded to the server; Selecting one of the log files as the current log file, aggregating the current log file according to the read aggregating rule, obtaining a aggregating result of the current log file, and uploading the aggregating result of the current log file to the server; Determine whether all the read log files have been uploaded to the server, and if not, continue to collect the next log file according to the read collection rule until all the read log files have been uploaded to the server; The aggregation rules include a splitting rule and a plurality of matching rules arranged in sequence; The current log file is aggregated according to the read aggregation rule to obtain the aggregation result of the current log file, including: splitting the current log file into log data according to the splitting rule; Selecting one of the log data as the current log data, and selecting one of the matching rules as the current matching rule according to the arrangement order of the matching rules; Match the current log data using the current matching rule, and determine whether the configuration of the current matching rule indicates to continue matching the current log data; If it is indicated to continue matching the current log data, determining whether there is a matching rule that does not match the current log data; If there is a matching rule that does not match the current log data, then according to the order of the matching rules, the next matching rule is selected as the current matching rule, and the current matching rule is continued to be used to match the current log data until there is no matching rule that does not match the current log data, and the matching result of the current log data is obtained; If the instruction is not to continue matching the current log data, the matching of the current log data is terminated, and it is determined whether there is the next log data that does not match the matching rule; If so, select the next log data as the current log data, continue to select a matching rule as the current matching rule according to the arrangement order of the matching rules, and use the current matching rule to match the current log data: If not, obtain the collection result of the current log file based on the matching results of all log data.

2. A distributed log collection method according to claim 1, characterized in that: The current matching rule configuration includes matching success configuration and matching failure configuration; Matching the current log data using the current matching rule includes: matching the current log data according to the regular expression in the current matching rule, and determining whether the regular expression is matched successfully; If yes, configure the current log data according to the successful matching configuration of the current matching rule to obtain the matching sub-result of the current matching rule; If not, configure the current log data according to the match failure configuration of the current matching rule; The matching result of the current log data is composed of the matching sub-results of all matching rules in the order of arrangement.

3. A distributed log collection method according to claim 2, characterized in that: Before matching the current log data according to the regular expression in the current matching rule, the method further includes: determining whether the regular expression in the current matching rule is empty; If not, matching the current log data according to the regular expression; If so, the current log data is configured according to the matching failure configuration of the current matching rule, and the next matching rule is selected as the current matching rule according to the arrangement order of the matching rules, and the current log data is continued to be matched using the current matching rule.

4. A distributed log collection method according to claim 2, characterized in that: The matching sub-result and the matching result are messages in json format; Configuring the current log data according to the successful matching configuration of the current matching rule, including: matching the current log data according to the regular expression, and obtaining the log character string in the current log data that matches the regular expression; According to the successful configuration of the match of the current matching rule, the matching of the current log data is ended, or a flag string is inserted into the tag of the message in the json format to mark the log string, the log string with the flag string is obtained or discarded, and the matching sub-result of the current matching rule is obtained; The log data is configured according to the match failure configuration of the current matching rule, including: according to the match failure configuration of the current matching rule, the matching of the current log data is ended, or a default value is inserted into the tag of the message in the json format.

5. A distributed log collection method according to claim 1, characterized in that: The content of the configuration template also includes a log path; reading the log file that has not been uploaded to the server includes: reading the log file that has not been uploaded to the server according to the log path.

6. A distributed log collection method according to claim 1, characterized in that: The collection rule also includes a separation mark; before selecting the next log data as the current log data, it also includes: setting the separation mark after the matching result of the current log data to separate the matching result of the current log data from the matching result of the next log data.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images