Log management method, platform, server and storage medium

Abstract

The application provides a log management method, platform, server and storage medium, which collects log data of logs generated by one or more applications during operation; if a log that meets the preset abnormal alarm condition is monitored, an alarm event is generated, and the alarm event is used to indicate that there is an abnormal log in the collected log; according to the alarm event, an alarm message is generated, and the alarm message is used to indicate the abnormal content in the abnormal log; according to the alarm message, the collected log data is retrieved to obtain the log data of the abnormal log. Through the method provided in this embodiment, logs can be managed uniformly, thereby improving log management efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of log management, and in particular to a log management method, platform, server and storage medium. Background Art

[0002] Logs are data that record system behaviors, application running status, and describe events. They are widely used in scenarios such as abnormal behavior troubleshooting, problem diagnosis, and analysis and statistics of application performance and usage. Logs are generally printed by applications on the target machine, and have the characteristics of large data volume, large data increments, and non-fixed data formats. With the continuous expansion of the application scenarios of logs, the management of logs is becoming more and more difficult. At present, for log management, most of them use specialized tools, such as: Elasticsearch servers for log collection and query, Logstash platforms and Kibana platforms, and open distributed tracing specifications (OpenTracing) for log link analysis. In actual applications, the inventors found that the above scheme has at least the following problems:

[0003] Current log management tools can only provide specific services and functions in fixed stages and scenarios in the log life cycle, and cannot manage logs in a unified manner, resulting in low log management efficiency. Summary of the invention

[0004] In a first aspect, the present application provides a log management method, comprising: collecting log data of logs generated during the operation of one or more applications;

[0005] If the monitored logs meet the preset abnormal alarm conditions, an alarm event is generated. The alarm event is used to indicate that there are abnormal logs in the collected logs;

[0006] Generate alarm information based on the alarm event, which is used to indicate the abnormal content in the abnormal log;

[0007] According to the alarm information, the collected logs are searched to obtain the log data of the abnormal logs.

[0008] Optionally, collect log data generated by one or more applications during operation, including:

[0009] Collect log data generated by one or more applications in real time during operation;

[0010] And the log data of the collected log is stored in the first storage device in real time.

[0011] Optionally, if a log that meets the preset abnormal alarm conditions is monitored, an alarm event is generated, including:

[0012] By consuming logs, determine whether there are logs that meet the preset abnormal alarm conditions;

[0013] If it exists, an alarm event is generated.

[0014] Optionally, based on the alarm information, the collected logs are retrieved to obtain log data of the abnormal logs, including:

[0015] Obtain reverse query credentials for abnormal logs based on alarm information;

[0016] Retrieve the log data of the abnormal log based on the reverse query credentials.

[0017] Optionally, the reverse lookup credential includes at least one of the following:

[0018] The storage path of the log data, the pointer offset of the log data, the timestamp of the log data, and the conditional predicate. The conditional predicate is the keyword key-value pair that initiates log retrieval.

[0019] Optionally, reverse query is performed based on the reverse query credentials, and the collected logs are retrieved to obtain log data of abnormal logs, including:

[0020] Retrieve the collected logs according to the conditional predicates;

[0021] If log data that satisfies at least one of the following items is detected: a storage path of the log data, a pointer offset of the log data, and a timestamp of the log data, then the log data corresponding to the abnormal log is obtained.

[0022] Optionally, the log data of the logs that have been consumed are written into the second storage device in a time series data format.

[0023] In a second aspect, the present application provides a log management platform, including: a log collection module, for collecting log data of logs generated during the operation of one or more applications;

[0024] The log monitoring module is used to generate an alarm event if a log that meets the preset abnormal alarm conditions is monitored. The alarm event is used to indicate that there is an abnormal log in the collected log;

[0025] An alarm analysis module is used to generate alarm information, and the alarm information is used to indicate abnormal content in the abnormal log;

[0026] The log retrieval module is used to retrieve the collected logs according to the alarm information and obtain the log data of the abnormal logs.

[0027] Optionally, when collecting log data of logs generated by one or more applications during operation, the log collection module is specifically used to:

[0028] Collect log data generated by one or more applications in real time during operation;

[0029] And the log data of the collected log is stored in the first storage device in real time.

[0030] Optionally, the log monitoring module generates an alarm event when monitoring the existence of a log that meets the preset abnormal alarm condition, and is also used to:

[0031] By consuming logs, determine whether there are logs that meet the preset abnormal alarm conditions;

[0032] If it exists, an alarm event is generated.

[0033] Optionally, when the alarm analysis module searches the collected logs according to the alarm information to obtain the log data of the abnormal log, it is specifically used to:

[0034] Obtain reverse query credentials for abnormal logs based on alarm information;

[0035] Retrieve the log data of the abnormal log based on the reverse query credentials.

[0036] Optionally, the reverse lookup credential includes at least one of the following:

[0037] The storage path of the log data, the pointer offset of the log data, the timestamp of the log data, and the conditional predicate. The conditional predicate is the keyword key-value pair that initiates log retrieval.

[0038] Optionally, when the log retrieval module performs reverse query according to the reverse query credentials, retrieves the collected logs, and obtains the log data of the abnormal log, it is specifically used to:

[0039] Retrieve the collected logs according to the conditional predicates;

[0040] If log data that satisfies at least one of the following items is detected: a storage path of the log data, a pointer offset of the log data, and a timestamp of the log data, then the log data corresponding to the abnormal log is obtained.

[0041] Optionally, the log monitoring module is further used to write the log data of the consumed logs into the second storage device in a time series data format.

[0042] In a third aspect, the present application provides a server, comprising: a memory for storing program instructions;

[0043] A processor is used to call and execute program instructions in a memory to perform a method as described in any one of the first aspects.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon; when the computer program is executed by a processor, the method as described in any one of the first aspects is implemented.

[0045] The present application provides a log management method, platform, server and storage medium, which collects log data of logs generated by one or more applications during operation; if a log that meets the preset abnormal alarm condition is monitored, an alarm event is generated, and the alarm event is used to indicate that there is an abnormal log in the collected log; according to the alarm event, an alarm message is generated, and the alarm message is used to indicate the abnormal content in the abnormal log; according to the alarm message, the collected log data is retrieved to obtain the log data of the abnormal log. Through the method provided in this embodiment, logs can be managed uniformly, thereby improving log management efficiency.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of the structure of a log management platform provided in one embodiment of the present application;

[0049] Figure 2 A schematic diagram of the structure of a log management platform provided in another embodiment of the present application;

[0050] Figure 3 An example diagram of the operation process of the log management platform provided in an embodiment of the present application;

[0051] Figure 4 A flow chart of a log management method provided in one embodiment of the present application;

[0052] Figure 5 A flow chart of a log management method provided by another embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of a server provided in one embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] In the description of the present application, it should be understood that the terms "upper", "upper", "lower", "front", "rear", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In the description of the invention, "plurality" means two or more, unless otherwise precisely and specifically specified.

[0056] The terms "first", "second", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these products or devices.

[0057] The description includes references to the accompanying drawings that form a part of the detailed description. The accompanying drawings show diagrams according to exemplary embodiments. These embodiments, which may also be referred to herein as "examples," are described in sufficient detail to enable those skilled in the art to practice the embodiments of the claimed subject matter described herein. The embodiments may be combined, other embodiments may be used, or structural, logical, and electrical changes may be made without departing from the scope and spirit of the claimed subject matter. It should be understood that the embodiments described herein are not intended to limit the scope of the subject matter, but rather to enable those skilled in the art to practice, make, and / or use the subject matter.

[0058] Below, some terms in this application are explained to facilitate understanding by those skilled in the art:

[0059] The time series database is used to store time series characteristic indicators and provide real-time time series indicator filtering and aggregation query services.

[0060] Data Map: (DataMap) is used to record data lineage and provide data positioning and traceability services.

[0061] Log model: (parser): What kind of log model the real-time analysis module uses to parse the input data.

[0062] Log consumption: read and write all data in order (FIFO).

[0063] Each log store (LogStore) has one or more partitions (Shard), and there are the following ways to read data:

[0064] (1) Can randomly fall into a certain partition;

[0065] (2) Logs can be read in batches from a specified partition in the order in which they are written to the partition;

[0066] (3) According to the time when the log is received, set the starting position (cursor) for batch reading of partition logs.

[0067] At present, there are many effective log processing solutions and storage analysis tools in the industry, such as Elasticsearch server, Logstash platform and Kibana platform for log collection and query, and OpenTracing for log link analysis, etc., but each tool has its own focus, providing services and functions for certain stages and scenarios in the log life cycle, and cannot manage logs in a unified way, which leads to restrictions on log usage scenarios and difficulty in expanding applications. Specifically, there are currently several problems with log management:

[0068] (1) When the number of logs is large, the data processing methods of the Elasticsearch server, Logstash platform, and Kibana platform cannot meet the needs of real-time log analysis;

[0069] (2) When the log volume is large, the Elasticsearch server, Logstash platform, and Kibana platform all have performance limitations and cannot meet the real-time log content monitoring and alarm requirements;

[0070] (3) The Elasticsearch server, Logstash platform, and Kibana platform all have design limitations and cannot meet the log original text association requirements of the alarm link.

[0071] Therefore, the current log management method can only provide specific services and functions in certain stages and scenarios in the log life cycle, and cannot manage logs in a unified manner, resulting in low log management efficiency and difficulty in meeting today's management needs for a large number of logs.

[0072] Based on the above problems, the embodiments of the present application provide a log management method, platform, server and storage medium. By implementing a unified architecture, complete log management functions including log collection, log monitoring and analysis, monitoring alarm and log retrieval are completed, breaking the barriers between the various stages in the log life cycle, and providing users with a unified log usage platform service. It solves the technical problem that the existing technology cannot manage logs in a unified manner, resulting in low log management efficiency, and meets the current management needs of a large number of logs.

[0073] Those skilled in the art will appreciate that the log management platform provided in this embodiment may be an electronic device with information processing capabilities, such as a smart phone, a tablet computer, a computer, or a network device, or a server with information processing capabilities.

[0074] Correspondingly, the log may be a log generated during the operation of an application on the electronic device, or a log generated during the operation of a server, and this embodiment does not impose any particular limitation on this.

[0075] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0076] The embodiment of the present application realizes a management platform for log collection, log monitoring, log retrieval and alarm analysis of log data, which can meet the log analysis needs of different applications and different devices and is a complete monitoring and alarm comprehensive service platform.

[0077] Figure 1 This is a schematic diagram of the structure of a log management platform provided in an embodiment of the present application. This embodiment does not strictly limit the names of the platform and its components, such as the log management platform 200 can be called a system log information collection and management system (Syslog Information Collection And Management System, SICMS).

[0078] Specifically, the log management platform 200 includes:

[0079] The log collection module 201 is used to collect log data generated by one or more application programs during operation;

[0080] The log monitoring module 202 is used to generate an alarm event if a log that meets a preset abnormal alarm condition is monitored, and the alarm event is used to indicate that an abnormal log exists in the collected log;

[0081] An alarm analysis module 203 is used to generate alarm information, where the alarm information is used to indicate abnormal content in the abnormal log;

[0082] The log retrieval module 204 is used to retrieve the collected logs according to the alarm information to obtain the log data of the abnormal log.

[0083] Optionally, when collecting log data of logs generated by one or more application programs during operation, the log collection module 201 is specifically used to:

[0084] Collect log data generated by one or more applications in real time during operation;

[0085] And the log data of the collected log is stored in the first storage device in real time.

[0086] Optionally, the log monitoring module 202 generates an alarm event when monitoring a log that meets a preset abnormal alarm condition, and is also used to:

[0087] By consuming logs, determine whether there are logs that meet the preset abnormal alarm conditions;

[0088] If it exists, an alarm event is generated.

[0089] Optionally, when the alarm analysis module 203 searches the collected logs according to the alarm information to obtain the log data of the abnormal log, it is specifically used to:

[0090] Obtain reverse query credentials for abnormal logs based on alarm information;

[0091] Retrieve the log data of the abnormal log based on the reverse query credentials.

[0092] Optionally, the reverse lookup credential includes at least one of the following:

[0093] The storage path of the log data, the pointer offset of the log data, the timestamp of the log data, and the conditional predicate. The conditional predicate is the keyword key-value pair that initiates log retrieval.

[0094] Optionally, when the log retrieval module 204 performs reverse query according to the reverse query credential, retrieves the collected logs, and obtains the log data of the abnormal log, it is specifically used to:

[0095] Retrieve the collected logs according to the conditional predicates;

[0096] If log data that satisfies at least one of the following items is detected: a storage path of the log data, a pointer offset of the log data, and a timestamp of the log data, then the log data corresponding to the abnormal log is obtained.

[0097] Optionally, the log monitoring module is further used to write the log data of the consumed logs into the second storage device in a time series data format.

[0098] A log management platform provided in an embodiment of the present application includes a log collection module, a log monitoring module, an alarm analysis module and a log retrieval module. Among them, the log collection module provides a log collection service, the log monitoring module provides a log monitoring service, the alarm analysis module provides an alarm analysis service and the log retrieval module provides a log retrieval service. Through the log management platform provided in this embodiment, logs can be managed uniformly, thereby improving log management efficiency.

[0099] exist Figure 1 Based on the log management platform 200, the log management platform 200 may also include other modules to jointly implement the log management function of the log management platform 200. Figure 2 Provide explanation.

[0100] Figure 2 This is a schematic diagram of the structure of a log management platform provided by another embodiment of the present application. Figure 2 As shown: the log management platform 200 also includes a log management system 205, through which the log management platform manages each module in an overall manner so that each module provides log collection service, log monitoring service, alarm analysis service and log retrieval service respectively.

[0101] The log management system 205 enables the log management platform 200 to complete complete log management functions including log collection, log monitoring, alarm analysis and log retrieval, breaking down the barriers between the various stages in the log life cycle and providing users with a unified log usage platform service.

[0102] In the log collection phase, the log collection module 201 is used to implement the log collection service. The log collection module 201 includes two major functions: real-time log collection and log data caching. In one embodiment, the log collection module 201 is implemented by a collector 206, wherein the collector 206 can be Filebeat. In another embodiment, the log collection module 201 is implemented by a distributed queue system 207 to implement the log data caching function, wherein the distributed queue system 207 can be a Kafka system.

[0103] Specifically, after the log collection module 201 collects the log data of the log during the application program operation, the log management system 205 records the attributes of the log data in the first data map 215 to obtain the processing link of the data collection. The log attributes may include at least one of the following: log path, IP address of the electronic device where the log is located, and log format.

[0104] Furthermore, the log collection aggregation point is specified through the log management system 205, that is, the above-mentioned Kafka, and the log collection program is obtained according to the data collection processing link, and the packaged log collection program is published to the electronic device where the log data source is located and Filebeat, and the log collection program is deployed in the electronic device where the log data source is located and Filebea, so as to enable Filebea to collect log data from the electronic device in real time.

[0105] In the subsequent log data collection process, the log data of the logs generated during the operation of the application program is collected from the electronic device by using Filebeat using the log collection program, and the collected log data is stored in Kafka in a preset queue form. Since Filebeat and Kafka are both open source systems, the log collection module 201 provided in the embodiment of the present application can meet the requirements of high capacity and scalability during the log collection and storage process.

[0106] In other embodiments, since the data in Kafka is typical unformatted data, and has characteristics such as data printing delay, data filling, and duplication, the data output to the downstream after processing is required to be well-formatted semi-structured data, and the statistical indicators generated need to be periodically aligned. Therefore, it is necessary to ensure effective identification of illegal data, format conversion with high fault tolerance, data alignment processing, etc. This function is implemented in the log monitoring stage. The specific content of the log monitoring stage is described below in conjunction with the log monitoring module 202.

[0107] In the log monitoring stage, the log monitoring module 202 is used to implement the log monitoring service. The log monitoring module 202 includes two major functions: real-time analysis and aggregation and index data storage. In one embodiment, the log analysis and aggregation module 208 is used to implement the real-time analysis and aggregation of logs, wherein the log analysis and aggregation module 208 can be a Flink module, and the Flink module includes a dynamic parsing and aggregation program developed based on Flink.

[0108] In one implementation, a log aggregation program is configured in the electronic device to implement the log monitoring function. The following describes a specific solution for the log monitoring module 202 to implement the log monitoring function in conjunction with the structure of the log monitoring module 202 .

[0109] First, the Flink module 208 accesses the Kafka system in the log collection module 201 through the first consumption interface (consume) 209. Specifically, the Flink module consumes the log data from the Kafka system 207 in real time and streaming through the first consumption interface 209 according to the preset log model and preset processing process.

[0110] Furthermore, the log management system 205 records the log consumption source, the aggregation processing procedure, and the processing result output in the first data map 215, thereby recording the processing link of the log monitoring data.

[0111] Furthermore, the log management system 205 reconfigures the logic of the log aggregation program in Flink according to the processing link, wherein the logic of configuring the Flink log aggregation program may include:

[0112] (1) Specify the upstream data source, that is, the Topic in Kafka;

[0113] (2) Processing link of log monitoring data;

[0114] (3) Specifying a downstream data storage path, wherein the downstream storage path may be a time series database (TSDB) 209. TSDB is a distributed non-structural database developed based on a distributed engine (ElasticSearch, Es).

[0115] Finally, after completing the logical configuration of the log aggregation program in the Flink module, the packaged log aggregation program is distributed and deployed in the electronic device where the log data source is located and in Flink.

[0116] In one implementation, the log monitoring module may further include: a network connection monitor (Argus) 214 .

[0117] During the process of the log monitoring module 202 providing log data monitoring services, since the logic of the log aggregation program is configured in the electronic device, the log monitoring module 202 can complete the real-time aggregation of log data in the electronic device through the log aggregation program, and according to the configured log aggregation program, write the aggregation results in the time series data format to the downstream time series database 209, and then use Argus to read the log data in TSDB to monitor the log data in the time series database 209.

[0118] Correspondingly, users can perform operations such as ad hoc time series indicator aggregation query in the time series database 209 through the time series database interface.

[0119] In the alarm analysis phase, the alarm analysis module 203 is used to implement the log alarm analysis service. Specifically, the alarm analysis module 203 is used to monitor and analyze the time series indicators.

[0120] In one embodiment, the alarm analysis module 203 may also include: a reverse query credential library 210 and an alarm engine 217. Specifically, the alarm engine 217 may adopt a Prometheus engine, and receive alarm events from the log monitoring module 202 through the alarm engine, wherein the log monitoring module 202 may send alarm events to the alarm engine through communication software such as email and text messages.

[0121] When an alarm event is received, the log management system triggers the alarm engine to obtain alarm information, where the alarm information may include: one or more of an indicator name, a filtering condition, a time range, and an aggregation method.

[0122] Furthermore, the reverse-check credential library 210 obtains the reverse-check credential of the abnormal log according to the alarm information, wherein the reverse-check credential can also be written into the reverse-check credential library 210, and the reverse-check credential ID corresponding to the reverse-check credential is obtained.

[0123] In one embodiment, the alarm information and the reverse query credential ID are sent to the operation and maintenance personnel, who locate the reverse query credential from the reverse query credential library 210 according to the alarm information and the reverse query credential ID, and then query the log data corresponding to the alarm event from the log original database Kafka according to the reverse query credential. The reverse query credential may include one or more of the storage path of the log data, the pointer offset of the log data, the timestamp of the log data, and the conditional predicate. The conditional predicate is a keyword key-value pair that initiates a query to the log original database Kafka.

[0124] In another implementation, the log retrieval module 204 may also perform a search based on the reverse query credentials and the alarm information to retrieve the log data corresponding to the alarm event from the log original database Kafka.

[0125] The log retrieval module 204 includes two major functions: real-time log retrieval and log data storage. Among them, real-time log retrieval can be implemented by data warehouse technology (Extract-Transform-Load, ETL), specifically, it can be implemented by the management platform 211, wherein the management platform 211 may include: Logstash platform; log data storage is implemented through the distributed engine 212, wherein the distributed engine 212 can be the ElasticSearch engine, referred to as Es.

[0126] Next, taking the Logstash platform and the Es engine as examples, the specific solution of the log retrieval module 204 to implement the log retrieval function is explained in combination with the structure of the log retrieval module 204.

[0127] In one implementation, the Logstash platform is connected to the log collection module 201 via the second consumption interface 213 .

[0128] First, the log events from Kafka are consumed in real time through the Logstash platform, and the log consumption source, consumption processing program, and processing result output are recorded in the second data map 216, thereby recording the log retrieval processing link;

[0129] Further, the log management system 205 reconfigures the logic of the log consumption program of the log management platform Logstash platform according to the log processing link, wherein configuring the logic of the log consumption program of the Logstash platform may include the following steps:

[0130] (1) Specify the upstream data source, that is, the Topic in Kafka;

[0131] (2) Log retrieval and processing link in the log consumption process;

[0132] (3) Specify a downstream data storage path, where the downstream storage path may be a distributed engine Es.

[0133] Finally, after completing the logical configuration of the log consumer program in the Logstash platform, the packaged log consumer program is distributed and deployed in the electronic device where the log data source is located.

[0134] Through the above steps, since the logic of the log consumption program is configured in the electronic device, the log retrieval module 204 can provide real-time retrieval services for log data in the electronic device through the log consumption program, and write the retrieval results to the downstream ES according to the configured log consumption program, thereby finally providing real-time log retrieval services.

[0135] In one embodiment, the log retrieval module 204 and the alarm analysis module 203 generate monitoring indicators and log original text respectively based on the same log data. The source and processing link relationship between the two data are represented by a semi-structured graph data. This type of data is a type of data with multiple retrieval dimensions and many-to-many relationships between points. Generally, multiple dimensions are used as query conditions to query data points on one or more connected graphs.

[0136] A log management platform provided in an embodiment of the present application includes a log management system, a log collection module, a log monitoring module, an alarm analysis module and a log retrieval module. The log management system manages each module in an overall manner so that each module provides a log collection service, a log monitoring service, an alarm analysis service and a log retrieval service respectively. Through the log management platform provided in this embodiment, logs can be managed uniformly, thereby improving log management efficiency.

[0137] Figure 3 This is an example diagram of the operation process of the log management platform provided in an embodiment of the present application. Figure 2 , Figure 3 The log retrieval service shown in FIG. 2 is provided by the log retrieval module 201, the log monitoring service is provided by the log monitoring module 202, the alarm analysis service is provided by the alarm analysis module 203, and the log retrieval service is provided by the log retrieval module 204. For ease of understanding, the specific operation process and principle of the log management platform are combined with Figure 4 The log management method embodiment provided is used for illustration, and this embodiment will not be repeated herein.

[0138] Figure 4 Flow chart of the log management method provided in one embodiment of the present application. The log management method provided in the embodiment of the present application can be executed by a computer or other devices, such as a server or other electronic device with information processing function, which is not particularly limited in this embodiment.

[0139] like Figure 4 As shown, the log management method includes the following steps:

[0140] S401: Collect log data of logs generated during the operation of one or more applications.

[0141] In actual application, one or more applications are electronic devices running log management programs. In one embodiment, the log data of the log generated by the electronic device during operation can be obtained after receiving the collection command issued by the user. In another embodiment, it can also be automatically obtained according to the operation of the electronic device, for example, when the electronic device starts to run, the log data starts to be collected, or when the electronic device stops running, the log data stops to be collected, which is not limited by this application.

[0142] For ease of understanding, the following Figure 3 To illustrate, in one implementation, Filebeat uses a log collection program to collect log data generated during the operation of the electronic device from the electronic device, and the collected log data is stored in Kafka in a preset queue format.

[0143] S402: If a log satisfying a preset abnormal alarm condition is detected, an alarm event is generated.

[0144] Continue to combine Figure 3 To illustrate, the log monitoring module 202 can complete the real-time aggregation of log data in the electronic device through a log aggregation program to monitor whether there is a log in the log data of the electronic device that meets the preset abnormal alarm condition.

[0145] In actual applications, the system log is monitored according to the type of electronic device where the log is obtained, using the keyword template corresponding to each electronic device. The log monitoring operation mainly classifies the log data. The log data of each log can be divided into the following categories: dangerous log data, security log data, and at least one of unknown log data.

[0146] In one embodiment, when the log data is dangerous log data, that is, the conditions that meet the preset abnormal alarm are monitored, and an alarm event is generated, wherein the alarm event is used to indicate that there are abnormal logs in the collected logs. In another embodiment, when the log data is safe log data or unknown log data, that is, the logs that meet the preset abnormal alarm conditions are not monitored, at this time, there is no need to generate an alarm event.

[0147] In one implementation, the alarm event may be sent via communication software such as email or text messages.

[0148] S403: Generate alarm information according to the alarm event.

[0149] Correspondingly, the above alarm event is received.

[0150] Among them, the alarm information is used to indicate the abnormal content in the abnormal log.

[0151] In actual applications, the alarm information may include: one or more of the indicator name, filtering conditions, time range, and aggregation method.

[0152] S404: According to the alarm information, the collected log data is searched to obtain the log data of the abnormal log.

[0153] Please continue to refer to Figure 3 When the alarm analysis module 203 receives the alarm information, in one implementation, the alarm information may be sent to an operation and maintenance personnel, who then queries the log data corresponding to the alarm event from the log original database Kafka according to the alarm information.

[0154] In another implementation, the log retrieval module 204 may also perform a search based on the alarm information to retrieve log data corresponding to the alarm event from the log original database Kafka.

[0155] A log management method provided by an embodiment of the present application first collects log data of logs generated by one or more applications during operation; if a log that meets a preset abnormal alarm condition is monitored, an alarm event is generated, and the alarm event is used to indicate that an abnormal log exists in the collected log; then, according to the alarm event, an alarm message is generated, and the alarm message is used to indicate the abnormal content in the abnormal log; finally, according to the alarm message, the collected log data is retrieved to obtain the log data of the abnormal log. Through the method provided by this embodiment, logs can be managed uniformly, thereby improving log management efficiency.

[0156] Figure 5 This is a flow chart of a log management method provided by another embodiment of the present application. Figure 5 As shown, the log management method provided in this embodiment may include the following steps:

[0157] S501. Collect log data of logs generated by one or more application programs during operation in real time, and store the collected log data in a first storage device in real time.

[0158] The steps of collecting log data generated by one or more application programs in real time during operation are as follows: Figure 4 Similar to S401, please refer to Figure 4 The embodiments provided will not be described in detail here.

[0159] In one implementation, the first storage device may be a Kafka system.

[0160] For more details, please refer to Figure 3 ,After Filebeat uses the log collection program to collect the log data of the logs generated during the operation of the electronic device, the collected log data is saved in Kafka in the form of a preset queue.

[0161] S502: Determine whether there is a log that meets the preset abnormal alarm condition through consumption logs;

[0162] If it exists, an alarm event is generated.

[0163] Please continue to refer to Figure 3 , the Flink module accesses the Kafka system in the log collection module 201 through the first consumption interface (consume) 209. Specifically, the Flink module consumes the log data from the Kafka system in real time and streaming according to the preset log model and preset processing process through the consumption interface 209. The consumed log data is stored in TSDB, and Argus is used to read the log data stored in TSDB to determine whether there are logs that meet the preset abnormal alarm conditions.

[0164] If it is determined that the log data in TSDB contains logs that meet the preset abnormal alarm conditions, an alarm event is generated.

[0165] Among them, the steps of generating alarm events are Figure 4 Similar to S402, please refer to Figure 4 The embodiments provided will not be described in detail here.

[0166] S503: Obtain alarm information according to the alarm event, and obtain reverse query credentials of the abnormal log according to the alarm information.

[0167] Among them, the process of obtaining alarm information according to the alarm event is Figure 4 Similar to S403, please refer to Figure 4 The embodiments provided will not be described in detail here.

[0168] In some embodiments, the reverse credential may include at least one of the following:

[0169] The storage path of the log data, the pointer offset of the log data, the timestamp of the log data, and the conditional predicate, where the conditional predicate is the keyword key-value pair that initiates log retrieval.

[0170] Combine the following Figure 3 The process of obtaining the reverse query credential of the abnormal log according to the alarm information is specifically described. The reverse query credential of the abnormal log is obtained according to the alarm information, wherein the reverse query credential can also be written into the reverse query credential library 210, and the reverse query credential ID corresponding to the reverse query credential is obtained.

[0171] S504: Search according to the reverse query credential to obtain log data of the abnormal log.

[0172] Please continue to refer to Figure 3 In one implementation, the alarm information and the reverse-check credential ID are sent to the operation and maintenance personnel, who locate the reverse-check credential from the reverse-check credential library 210 based on the alarm information and the reverse-check credential ID, and then query the log data corresponding to the alarm event from the log original database Kafka based on the reverse-check credential.

[0173] In another implementation, the log retrieval module 204 may also perform a search based on the reverse query credentials and the alarm information to retrieve the log data corresponding to the alarm event from the log original database Kafka.

[0174] Specifically, retrieving the log data corresponding to the alarm event from the log original database Kafka may further include the following steps:

[0175] (1) According to the condition predicate, the collected logs are retrieved in Kafka;

[0176] (2) If log data that satisfies at least one of the following items is detected: a storage path of the log data, a pointer offset of the log data, and a timestamp of the log data, then the log data corresponding to the abnormal log is obtained.

[0177] A log management method provided by an embodiment of the present application first collects log data of logs generated by one or more applications during operation in real time, and stores the collected log data in real time in a first storage device; by consuming logs, it is determined whether there are logs that meet the preset abnormal alarm conditions, and if so, an alarm event is generated; alarm information is obtained according to the alarm event, and reverse query credentials of the abnormal log are obtained according to the alarm information; finally, the log data of the abnormal log is retrieved according to the reverse query credentials. Through the method provided by this embodiment, logs can be managed uniformly, thereby improving log management efficiency. In addition, the above method can provide log collection services, log monitoring services, alarm analysis services, and log retrieval services, further improving log management efficiency.

[0178] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. Figure 6 The server 60 includes a processing component 601, which further includes one or more processors, and a memory resource represented by a memory 602 for storing instructions executable by the processing component 601, such as an application. The application stored in the memory 602 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 601 is configured to execute instructions to perform such as Figure 4 , Figure 5 For the method embodiment shown, please refer to the description in the above method embodiment for details, which will not be repeated here.

[0179] The server 60 may also include a wired or wireless network interface 603 configured to connect the server 60 to a network, and an input / output (I / O) interface 604. The server 60 may operate based on an operating system stored in the memory 602, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0180] It can be understood by those skilled in the art that Figure 6 The structure of the server shown in the figure does not constitute a limitation on the present server, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the optimization method of any of the above embodiments.

[0182] In the above-mentioned embodiments, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0183] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0184] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0185] It should be understood that the above processor can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), etc. A general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0186] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0187] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk, etc. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0188] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A log management method, characterized in that: include: Record the log path, the IP address of the electronic device where the log is located, and the log format in the first data map to obtain a processing link for data collection; Obtaining a log collection program according to the processing link; collecting log data of logs generated by one or more application programs during operation through the log collection program; Record the log consumption source, aggregation processing program, and processing result output in the first data map to obtain the processing link of the log monitoring data; Reconfigure the log aggregation program logic according to the processing link, aggregate the log data through the reconfigured log aggregation program, express the aggregated result in a time series data format, and classify the log data expressed in the time series data format using a keyword template to determine whether there are logs that meet the abnormal alarm conditions; the log data includes: at least one of dangerous log data, security log data and unknown log data; If a log that meets the preset abnormal alarm condition is monitored, an alarm event is generated, and the alarm event is used to indicate that there is an abnormal log in the collected log; Generate alarm information according to the alarm event, wherein the alarm information is used to indicate abnormal content in the abnormal log; According to the alarm information, the collected log data is retrieved to obtain the log data of the abnormal log; The retrieval of the collected log data includes: recording the log consumption source, consumption processing program, and processing result output in the second data map, obtaining a log retrieval processing link, reconfiguring the logic of the log consumption program according to the log processing link, and performing log retrieval according to the configured log consumption program.

2. The method according to claim 1, characterized in that: Collect log data generated by one or more applications during operation, including: Collect log data generated by one or more applications in real time during operation; And the collected log data of the log is stored in the first storage device in real time.

3. The method according to claim 2, characterized in that If the monitoring finds logs that meet the preset abnormal alarm conditions, an alarm event is generated, including: By consuming the logs, determining whether there are logs that meet the preset abnormal alarm conditions; If so, the alarm event is generated.

4. The method according to claim 3, characterized in that The step of retrieving the collected logs according to the alarm information to obtain the log data of the abnormal log includes: Obtaining reverse query credentials for the abnormal log according to the alarm information; Retrieve according to the reverse query credentials to obtain the log data of the abnormal log.

5. The method according to claim 4, characterized in that The reverse query credential includes at least one of the following: The storage path of the log data, the pointer offset of the log data, the timestamp of the log data and the conditional predicate, where the conditional predicate is a keyword key-value pair for initiating log retrieval.

6. The method according to claim 5, characterized in that The reverse query is performed according to the reverse query credentials, and the collected logs are retrieved to obtain the log data of the abnormal log, including: Retrieving the collected logs according to the condition predicate; If it is detected that at least one of the following items is satisfied: the storage path of the log data, the pointer offset of the log data, and the log data of the timestamp of the log data, the log data corresponding to the abnormal log is obtained.

7. The method according to any one of claims 3 to 6, characterized in that: Also includes: The log data of the logs that have been consumed are written into the second storage device in a time series data format.

8. A log management platform, characterized in that: include: A log collection module, used to record the log path, the IP address of the electronic device where the log is located, and the log format in the first data map to obtain a processing link for data collection; Obtaining a log collection program according to the processing link; Collect log data generated by one or more applications during operation through a log collection program; A log monitoring module, used to record the log consumption source, the aggregation processing program, and the processing result output in the first data map to obtain the processing link of the log monitoring data; Reconfigure the log aggregation program logic according to the processing link, aggregate the log data through the reconfigured log aggregation program, express the aggregated result in a time series data format, and classify the log data expressed in the time series data format using a keyword template to determine whether there is a log that meets the abnormal alarm condition; the log data includes: at least one of dangerous log data, security log data and unknown log data; if the log that meets the preset abnormal alarm condition is monitored, an alarm event is generated, and the alarm event is used to indicate that there is an abnormal log in the collected log; An alarm analysis module, used to generate alarm information, wherein the alarm information is used to indicate abnormal content in the abnormal log; A log retrieval module, used to retrieve the collected logs according to the alarm information to obtain the log data of the abnormal log; The log retrieval module is specifically used to record the log consumption source, consumption processing program, and processing result output in the second data map, obtain the log retrieval processing link, reconfigure the logic of the log consumption program according to the log processing link, and perform log retrieval according to the configured log consumption program.

9. A server, characterized in that: include: A memory for storing program instructions; A processor, configured to call and execute program instructions in the memory to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the log management method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the log management method according to any one of claims 1 to 7.

Images