Log collection and log monitoring method and device
Patent Information
- Application Number
- CN202110862175.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-29
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-07-29
AI Technical Summary
[0005]针对上述问题,本发明提供了一种日志采集及日志监控方法、装置,有效解决现有日志采集及日志监控运维成本过高的技术问题
[0046]本发明提供的日志采集及日志监控方法、装置,实现了基于flink的流计算任务在on yarn模式和on kubernetes模式下统一灵活的日志采集方式,通过这种方式可以根据的自己的需要定义自己的日志格式和添加业务需要的元数据信息,且无需要运维人员单独维护类似logstash的组件,极大的降低了运维成本。
Smart Images

Figure CN113590443B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a log collection and log monitoring method and device. Background Art
[0002] Currently, the most used open source log collection and monitoring solution is
[0003] Elasticsearch+Logstash+Kibana, referred to as ELK, where Elasticsearch is a Lucene-based search server that provides a distributed multi-user full-text search engine; Logstash is a platform for the transmission, processing, management, and search of application logs and events;
[0004] Kibana is a web interface for log analysis provided by Logstash and ElasticSearch. Although ELK can solve the problem of log collection and monitoring, if you want to collect logs on yarn (a resource scheduling platform that is responsible for providing computing programs with the resources required for computing, equivalent to a distributed operating system), you need to deploy logstash on each node of yarn, which will bring two problems: 1) logstash will always occupy node resources; 2) when the cluster is expanded, logstash needs to be installed on the new node, which greatly increases the operation and maintenance costs. Summary of the invention
[0005] In view of the above problems, the present invention provides a log collection and log monitoring method and device, which effectively solves the technical problem of high operation and maintenance costs of existing log collection and log monitoring.
[0006] The technical solution provided by the present invention is as follows:
[0007] In one aspect, the present invention provides a log collection and log monitoring method, comprising:
[0008] Customize log4j appender to collect flink logs and write them to kafka;
[0009] Rewrite the flink client configuration to customize the log4j appender;
[0010] Based on flink sql, the flink logs in kafka are written to elasticsearch and the abnormal logs are matched based on the sql operator to realize the collection and monitoring of flink logs.
[0011] Further preferably, before collecting the flink logs based on the custom log4j appender and writing them into kafka, a step of customizing the log4j appender is included; the customizing step of the log4j appender includes:
[0012] Inherit the AppenderSkeleton class to define the configuration parameters of Kafka;
[0013] Parse the Kafka configuration parameters based on the activateOptions method and initialize the Kafka producer;
[0014] Based on the append method, the task parameters are parsed from the environment variables and written into Kafka;
[0015] Use the Maven plug-in to build a jar package containing the Kafka class and a jar package that does not contain the Kafka class to configure the log4j appender.
[0016] Further preferably, in the rewriting of the flink client to configure a custom log4j appender, the step of rewriting the flinkclient includes:
[0017] Rewrite the YarnClusterDescriptor class in the on yarn mode, upload the jar packages containing the Kafka class and the jar packages excluding the Kafka class in the log4j appender to HDFS in the shipfile mode, and add the jar packages excluding the Kafka class to the classpath directory;
[0018] Build the image in on kubernetes mode, add the jar package that does not contain the kafka class in the log4j appender to the flink lib directory, and add the jar package that contains the kafka class in the log4j appender to the flink opt directory;
[0019] Add environment variables TASK_ID and DTSTACK_APPENDER_JAR. The environment variable DTSTACK_APPENDER_JAR is used to specify the path of the jar package containing the kafka class in the log4j appender.
[0020] Configure log4j.properties and add the custom log4j appender with the above configuration.
[0021] Further preferably, the step of writing the flink log in kafka to elasticsearch based on flink sql and matching the abnormal log based on the sql operator includes:
[0022] The log information of Flink written to Kafka is read in real time through the Kafka source table defined in Flink SQL;
[0023] The exception log is captured by matching the REGEXP operator with the predefined exception log regular expression and output to the alert_table table, which is created in the MySQL database.
[0024] Read the alert_table table data through the DataStack custom alarm interface to implement abnormal log alarm.
[0025] In another aspect, the present invention provides a log collection and log monitoring device, comprising:
[0026] The log collection module is used to customize the log4j appender to collect Flink logs and write them to Kafka;
[0027] Configuration rewrite module, used to rewrite flink client configuration custom log4j appender;
[0028] The log monitoring module is used to write the Flink logs in Kafka to Elasticsearch based on Flink SQL and match the abnormal logs based on SQL operators to realize the collection and monitoring of Flink logs.
[0029] Further preferably, the log collection and log monitoring device also includes a log4j appender configuration module, including:
[0030] The parameter definition unit is used to inherit the AppenderSkeleton class to define the configuration parameters of Kafka;
[0031] Initialization unit, used to parse Kafka configuration parameters and initialize Kafkaproducer based on the activateOptions method;
[0032] The task parameter parsing unit is used to parse the task parameters from the environment variables based on the append method and write them to Kafka;
[0033] Configuration unit, used to use the Maven plugin to build a jar package containing the Kafka class and a jar package without the Kafka class to configure the log4j appender.
[0034] Further preferably, the configuration rewriting module includes:
[0035] On yarn mode configuration unit, rewrite the class YarnClusterDescriptor in on yarn mode, upload the jar package containing the Kafka class and the jar package without the Kafka class in the log4j appender to hdfs in the shipfile mode, and add the jar package without the Kafka class to the classpath directory;
[0036] On kubernetes mode configuration unit, build the image in on kubernetes mode, add the jar package that does not contain the kafka class in the log4jappender to the flink lib directory, and add the jar package that contains the kafka class in the log4j appender to the flink opt directory;
[0037] Variable configuration unit, used to increase the environment variables TASK_ID and
[0038] DTSTACK_APPENDER_JAR, where the environment variable DTSTACK_APPENDER_JAR is used to specify the path of the jar package containing the kafka class in the log4jappender;
[0039] The log4j.properties configuration unit is used to configure log4j.properties and add a custom log4j appender for the configuration.
[0040] Further preferably, the log monitoring module includes:
[0041] The log information reading unit is used to read the log information written to Flink in Kafka in real time through the Kafka source table defined in Flink SQL;
[0042] The exception log capture unit is used to capture the exception log through the REGEXP operator and the predefined exception log matching regular expression, and output it to the alert_table table, where the alert_table table is created in the MySQL database;
[0043] The exception log alarm unit is used to read the alert_table table data through the data stack custom alarm interface to implement exception log alarm.
[0044] On the other hand, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned log collection and log monitoring method when executing the computer program.
[0045] On the other hand, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned log collection and log monitoring method are implemented.
[0046] The log collection and log monitoring method and device provided by the present invention realize a unified and flexible log collection method for Flink-based stream computing tasks in the on YARN mode and the on Kubernetes mode. In this way, the user can define his own log format and add metadata information required by the business according to his own needs, and there is no need for operation and maintenance personnel to separately maintain components similar to Logstash, which greatly reduces the operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The preferred implementation scheme will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and their implementation methods.
[0048] Figure 1 This is a flow chart of the log collection and log monitoring method in the present invention;
[0049] Figure 2 It is a structural diagram of the log collection and log monitoring device in the present invention.
[0050] Reference numerals:
[0051] 100 - log collection and log monitoring device, 110 - log collection module, 120 - configuration rewriting module, 130 - log monitoring module. DETAILED DESCRIPTION
[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.
[0053] A first embodiment of the present invention is a log collection and log monitoring method, such as Figure 1 As shown, including:
[0054] S10 customizes the log4j appender to collect flink logs and write them to kafka;
[0055] S20 rewrites flink client to configure a custom log4j appender;
[0056] Based on Flink SQL, S30 writes Flink logs in Kafka to ElasticSearch and matches abnormal logs based on SQL operators to collect and monitor Flink logs.
[0057] In this embodiment, before collecting and monitoring logs, the log4j appender is customized based on the Java programming language. The customization process includes:
[0058] S11 inherits the AppenderSkeleton class and defines the relevant configuration parameters of Kafka (including but not limited to broker, topic, etc.).
[0059] S12 rewrites the activateOptions method to parse the Kafka configuration parameters and initialize the Kafka producer. In this process, in order to avoid class conflicts with the Flink Kafka connector, the Flink child classloader is used to load the Kafka appender when initializing the Kafkaproducer.
[0060] S13 rewrites the append method, parses taskId, applicationId and other task-related parameters from environment variables (including but not limited to taskId, applicationId, taskName, etc.) and writes them to Kafka. Here, writing to Kafka adopts asynchronous writing mode and uses try catch to catch exceptions, so that even if writing fails, it will not affect the Flink log stream.
[0061] S14 uses the Maven plug-in to build a jar package containing the Kafka class (dt-appender-kafka.jar) and a jar package that does not contain the Kafka class (dt-appender-lib.jar) to complete the customization of the log4jappender.
[0062] In step S20, in which the flink client is rewritten to configure a custom log4j appender, the steps of rewriting the flink client based on the Java programming language include:
[0063] S21 rewrites the YarnClusterDescriptor class in the on yarn mode, uploads the jar package (dt-appender-kafka.jar) containing the Kafka class and the jar package (dt-appender-lib.jar) without the Kafka class in the custom log4jappender to hdfs (hadoop distributed file system) as shipfile, and adds the jar package (dt-appender-lib.jar) without the Kafka class to the classpath directory;
[0064] S22 builds an image in on kubernetes mode, adds the jar package (dt-appender-lib.jar) that does not contain the kafka class in the log4j appender to the flink lib directory, and adds the jar package (dt-appender-kafka.jar) that contains the kafka class in the log4j appender to the flink opt directory;
[0065] S23 adds environment variables TASK_ID and DTSTACK_APPENDER_JAR, where the environment variable DTSTACK_APPENDER_JAR is used to specify the path of the jar package containing the kafka class in the log4j appender;
[0066] S24 configures log4j.properties and adds a custom log4j appender.
[0067] In step S30, the flink log in kafka is written to elasticsearch based on flink sql and the abnormal log is matched based on the sql operator, including:
[0068] S31 reads the log information written to Flink in Kafka in real time through the Kafka source table defined in Flink SQL;
[0069] S32 uses the REGEXP operator and predefined exception log matching regular expressions to capture exception logs and output them to the alert_table table. The alert_table table is created in the MySQL database and is used to save matching exception logs.
[0070] S33 reads the alert_table table data through the data stack custom alarm interface to implement abnormal log alarm.
[0071] In actual applications, after the web application layer submits tasks to the flink client, the log4j properties are configured in the flink client, and environment variables such as taskId are set. When the flink client submits tasks to yarn, the custom log4j appender jar is uploaded to the yarn container and the task is submitted to k8s (kubernetes). The jobmanger custom log4jappender in the yarn container collects and writes flink logs to kafka, and the jobmanger custom log4j appender in the pod in kubernetes collects and writes flink logs to kafka.
[0072] Another embodiment of the present invention is a log collection and log monitoring device, such as Figure 2 As shown, based on the Java programming language, the log collection and log monitoring device includes: a log collection module, which is used to customize the log4j appender to collect and write the flink logs into kafka; a configuration rewriting module, which is used to rewrite the flink client configuration to customize the log4j appender; a log monitoring module, which is used to write the flink logs in kafka into elasticsearch based on flink sql and match the abnormal logs based on the sql operator to realize the collection and monitoring of the flink logs.
[0073] In addition, the log collection and log monitoring device also includes a log4j appender configuration module, including: a parameter definition unit, which is used to inherit the AppenderSkeleton class and define the configuration parameters of kafka (including but not limited to broker, topic, etc.); an initialization unit, which is used to parse the kafka configuration parameters based on the activateOptions method and initialize the kafkaproducer; a task parameter parsing unit, which is used to parse the task parameters (including but not limited to taskId, applicationId, taskName, etc.) from the environment variables based on the append method and write them into kafka; a configuration unit, which is used to use the maven plug-in to build a jar package containing the kafka class and a jar package not containing the kafka class, and configure the log4jappender.
[0074] Specifically, in the initialization unit, rewrite the activateOptions method, parse the kafka configuration parameters and initialize the kafka producer. In this process, in order to avoid class conflicts with the flink kafka connector, use the flink child classloader to load the kafkaappender when initializing the kafka producer. Rewrite the append method in the task parameter parsing unit, parse the taskId, applicationId and other task-related parameters from the environment variables and write them to kafka. Here, writing to kafka uses asynchronous writing and uses try catch to capture exceptions, so that even if the writing fails, it will not affect the flink log stream. In the configuration unit, use the maven plug-in to build a jar package containing the kafka class (dt-appender-kafka.jar) and a jar package that does not contain the kafka class (dt-appender-lib.jar) to complete the customization of the log4j appender.
[0075] The configuration rewrite module includes: on yarn mode configuration unit, which rewrites the YarnClusterDescriptor class in on yarn mode, uploads the jar package (dt-appender-kafka.jar) containing the kafka class and the jar package (dt-appender-lib.jar) without the kafka class in the custom log4j appender to hdfs (hadoop distributed file system) as shipfile, and adds the jar package (dt-appender-lib.jar) without the kafka class to the classpath directory; on kubernetes mode configuration unit, which builds an image in on kubernetes mode, adds the jar package (dt-appender-lib.jar) without the kafka class in the log4jappender to the flink lib directory, and adds the jar package (dt-appender-kafka.jar) containing the kafka class in the log4j appender to the flink opt directory; variable configuration unit, which is used to increase the environment variables TASK_ID and
[0076] DTSTACK_APPENDER_JAR, where the environment variable DTSTACK_APPENDER_JAR is used to specify the path of the jar package containing the kafka class in the log4jappender; the log4j.properties configuration unit is used to configure log4j.properties and newly configure the custom log4j appender.
[0077] The log monitoring module includes: a log information reading unit, which is used to read the log information of Flink written to Kafka in real time through the Kafkasource table defined in Flink SQL; an exception log capture unit, which is used to capture exception logs through the REGEXP operator and the predefined exception log matching regular expression, and output them to the alert_table table. The alert_table table is created in the MySQL database; an exception log alarm unit, which is used to read the alert_table table data through the Data Stack custom alarm interface to implement exception log alarms.
[0078] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present invention. It should be pointed out that ordinary relevant personnel in this technical field can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.
Claims
1. A log collection and log monitoring method, It is characterized in that include: Customize log4j appender to collect flink logs and write them to kafka; Rewrite the flink client configuration to customize the log4j appender; Based on flink sql, flink logs in kafka are written to elasticsearch and abnormal logs are matched based on sql operators to collect and monitor flink logs. Before the custom log4j appender collects the flink logs and writes them into kafka, the steps of customizing the log4j appender are included; the customizing steps of the log4j appender include: Inherit the AppenderSkeleton class to define the configuration parameters of Kafka; Parse the Kafka configuration parameters based on the activateOptions method and initialize the Kafka producer; Based on the append method, the task parameters are parsed from the environment variables and written into Kafka; Use the Maven plug-in to build a jar package containing the Kafka class and a jar package that does not contain the Kafka class, and configure the log4jappender; In the above-mentioned rewriting of the flink client configuration custom log4j appender, the steps of rewriting the flink client include: Rewrite the YarnClusterDescriptor class in the on yarn mode, upload the jar packages containing the Kafka class and the jar packages excluding the Kafka class in the log4j appender to HDFS in the shipfile mode, and add the jar packages excluding the Kafka class to the classpath directory; Build the image in on kubernetes mode, add the jar package that does not contain the kafka class in the log4j appender to the flink lib directory, and add the jar package that contains the kafka class in the log4j appender to the flink opt directory; Add environment variables TASK_ID and DTSTACK_APPENDER_JAR. The environment variable DTSTACK_APPENDER_JAR is used to specify the path of the jar package containing the kafka class in the log4j appender. Configure log4j.properties and add the custom log4j appender with the above configuration.
2. The log collection and log monitoring method according to claim 1, It is characterized in that In the method of writing the flink log in kafka to elasticsearch based on flink sql and matching the abnormal log based on sql operator, the following is included: The log information of Flink written to Kafka is read in real time through the Kafka source table defined in Flink SQL; The exception log is captured by matching the REGEXP operator with the predefined exception log regular expression and output to the alert_table table, which is created in the MySQL database. Read the alert_table table data through the DataStack custom alarm interface to implement abnormal log alarm.
3. A log collection and log monitoring device, It is characterized in that include: The log collection module is used to customize the log4j appender to collect Flink logs and write them to Kafka; Configuration rewrite module, used to rewrite flink client configuration custom log4j appender; The log monitoring module is used to write the flink logs in kafka to elasticsearch based on flink sql and match the abnormal logs based on sql operators to realize the collection and monitoring of flink logs; The log collection and log monitoring device also includes a log4j appender configuration module, including: The parameter definition unit is used to inherit the AppenderSkeleton class to define the configuration parameters of Kafka; Initialization unit, used to parse Kafka configuration parameters and initialize Kafka producer based on the activateOptions method; The task parameter parsing unit is used to parse the task parameters from the environment variables based on the append method and write them to Kafka; The configuration unit is used to use the Maven plugin to build a jar package containing the Kafka class and a jar package without the Kafka class to configure the log4j appender; The configuration rewriting module includes: On yarn mode configuration unit, rewrite the class YarnClusterDescriptor in on yarn mode, upload the jar package containing the Kafka class and the jar package without the Kafka class in the log4jappender to hdfs in the shipfile mode, and add the jar package without the Kafka class to the classpath directory; On kubernetes mode configuration unit, build the image in on kubernetes mode, add the jar package that does not contain the kafka class in the log4jappender to the flink lib directory, and add the jar package that contains the kafka class in the log4j appender to the flink opt directory; The variable configuration unit is used to add the environment variables TASK_ID and DTSTACK_APPENDER_JAR. The environment variable DTSTACK_APPENDER_JAR is used to specify the path of the jar package containing the kafka class in the log4j appender. The log4j.properties configuration unit is used to configure log4j.properties and add a custom log4j appender for the configuration.
4. The log collection and log monitoring device according to claim 3, It is characterized in that The log monitoring module includes: The log information reading unit is used to read the log information written to Flink in Kafka in real time through the Kafka source table defined in Flink SQL; The exception log capture unit is used to capture the exception log through the REGEXP operator and the predefined exception log matching regular expression, and output it to the alert_table table, where the alert_table table is created in the MySQL database; The exception log alarm unit is used to read the alert_table table data through the data stack custom alarm interface to implement exception log alarm.
