Monitoring system, method, electronic device and readable medium

Abstract

The embodiment of the present invention provides a monitoring system, method, electronic device and readable medium, including a Filebeat component, a Zookeeper node and a background monitoring management program; wherein the Filebeat component is used to send the running status information of the Filebeat component to the Zookeeper node; wherein the running status information of the Filebeat component includes the rate at which the Filebeat component collects logs locally and the status information of the Filebeat component; the Zookeeper node is used to push the running status information of the Filebeat component to the background monitoring management program to monitor the running status of the Filebeat component. In the embodiment of the present invention, based on the publish-subscribe technology of zookeepr, monitoring is added to the link of log collection, which not only saves labor costs and shortens the troubleshooting time cycle compared to the manual execution processing method, but also can quickly locate the problem in the log collection process.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a monitoring system, a monitoring method, an electronic device and a computer-readable medium. Background Art

[0002] As the scale of information systems gradually expands, more and more hosts, application systems, development languages, etc. are added to the system network, and log security management becomes more and more complicated. System services usually use the mainstream ELK log collection and storage solution. Each log must be processed by multiple components such as filebeat, kafka, and logstash before it can be finally written into the Elasticsearch component. If any of the components in the middle has a problem, log collection will fail.

[0003] Currently, when log collection fails, such as when logs are lost or backlogged, staff usually log in to the corresponding server to troubleshoot each link of log collection. However, the above troubleshooting method not only takes a long time to troubleshoot, consumes a lot of manpower costs, but also makes it difficult to find the root cause of the problem. Summary of the invention

[0004] The embodiments of the present invention provide a monitoring system, method, electronic device and computer-readable storage medium to solve the problem that the cause of log collection failure is difficult to find out.

[0005] The embodiment of the present invention discloses a monitoring system, including a Filebeat component, a Zookeeper node and a background monitoring management program; wherein:

[0006] The Filebeat component is used to send the running status information of the Filebeat component to the Zookeeper node; wherein the running status information of the Filebeat component includes the rate at which the Filebeat component collects logs from the local and the status information of the Filebeat component;

[0007] The Zookeeper node is used to push the running status information of the Filebeat component to the background monitoring management program to monitor the running status of the Filebeat component.

[0008] Optionally, the Ec_gostash component is also included;

[0009] The Ec_gostash component is used to send the running status information of the Ec_gostash component to the Zookeeper node; wherein the running status information of the Ec_gostash component includes the rate at which the Ec_gostash component processes logs and the status information of the Filebeat component;

[0010] The Zookeeper node is also used to push the running status information of the Ec_gostash component to the background monitoring management program to monitor the running status of the Ec_gostash component.

[0011] Optionally, Kafka components and Elasticsearch components are also included;

[0012] The Kafka component is used to classify and cache the logs collected by the Filebeat component;

[0013] The Ec_gostash component is also used to filter the logs cached in the Kafka component and write the filtered logs into the Elasticsearch component for storage.

[0014] Optionally, the Ec_gostash component includes a Filter module;

[0015] The Ec_gostash component is also used for a custom field field, the field field maps the corresponding field in the log cached in the Kafka component, and the field field is input into the Filter module to filter the log corresponding to the field field.

[0016] Optionally,

[0017] The Ec_gostash component is further used to record the filtered diaries and the filtering reasons thereof by using the Filter module, and write the filtered diaries and the filtering reasons into a local computer.

[0018] An embodiment of the present invention discloses a monitoring method, comprising:

[0019] Sending the running status information of the Filebeat component to the Zookeeper node through the Filebeat component; wherein the running status information of the Filebeat component includes the rate at which the Filebeat component collects logs from the local and the status information of the Filebeat component;

[0020] The running status information of the Filebeat component is pushed to the background monitoring management program through the Zookeeper node to monitor the running status of the Filebeat component.

[0021] Optionally, it also includes:

[0022] Sending the running status information of the Ec_gostash component to the Zookeeper node through the Ec_gostash component; wherein the running status information of the Ec_gostash component includes the rate at which the Ec_gostash component processes logs and the status information of the Filebeat component;

[0023] The running status information of the Ec_gostash component is pushed to the background monitoring management program through the Zookeeper node to monitor the running status of the Ec_gostash component.

[0024] Optionally, it also includes:

[0025] Classify and cache the logs collected by the Filebeat component through the Kafka component;

[0026] The logs cached in the Kafka component are filtered by the Ec_gostash component, and the filtered logs are written into the Elasticsearch component for storage.

[0027] Optionally, the Ec_gostash component includes a Filter module, and the filtering process of the logs cached in the Kafka component by the Ec_gostash component includes:

[0028] Through the field field customized by the Ec_gostash component, the field field maps the corresponding field in the log cached in the Kafka component, and the field field is input into the Filter module to filter the log corresponding to the field field.

[0029] Optionally, it also includes:

[0030] The Ec_gostash component uses the Filter module to record the filtered diaries and the reasons for filtering, and writes the filtered diaries and the reasons for filtering to the local computer.

[0031] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0032] The memory is used to store computer programs;

[0033] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0034] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0035] The embodiment of the present invention includes the following advantages: the Filebeat component sends the running status information of the Filebeat component to the Zookeeper node; wherein, the running status information of the Filebeat component includes the rate at which the Filebeat component collects logs locally and the status information of the Filebeat component; the Zookeeper node pushes the running status information of the Filebeat component to the background monitoring management program to monitor the running status of the Filebeat component. In the embodiment of the present invention, based on the publish-subscribe technology of zookeepr, monitoring is added to the link of log collection, which not only saves labor costs and shortens the troubleshooting time cycle compared to the manual execution processing method, but also can quickly locate the problem in the log collection process.

[0036] At the same time, while solving the problem of log collection failure, it can reduce the occupancy rate of server resources, thereby improving the efficiency of log collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of the structure of a monitoring system provided in an embodiment of the present invention;

[0038] Figure 2 It is a structural diagram of a Filebeat component provided in an embodiment of the present invention;

[0039] Figure 3 It is a structural diagram of an Ec_gostash component provided in an embodiment of the present invention;

[0040] Figure 4 is a structural diagram of a Filter module provided in an embodiment of the present invention;

[0041] Figure 5 is a flowchart of a monitoring method provided in an embodiment of the present invention;

[0042] Figure 6 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0043] Figure 7 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Reference Figure 1 , shows a structural schematic diagram of a monitoring system provided in an embodiment of the present invention, the monitoring system includes a Filebeat component, a Zookeeper node and a background monitoring management program; wherein the Filebeat component is used to send the running status information of the Filebeat component to the Zookeeper node; wherein the running status information of the Filebeat component includes the rate at which the Filebeat component collects logs from the local and the status information of the Filebeat component; the Zookeeper node is used to push the running status information of the Filebeat component to the background monitoring management program to monitor the running status of the Filebeat component.

[0046] Among them, when log collection fails, such as log loss or log backlog, the components in the ELK log collection and storage system will also occupy a large amount of server resources, reducing the efficiency of log collection.

[0047] The Filebeat component is a lightweight log collector for forwarding and centralizing log data. It collects log content in an incremental manner and pushes the logs to the Kafka queue for buffering. The Filebeat component in the embodiment of the present invention is a secondary development component that can calculate the efficiency of collecting logs locally while collecting logs.

[0048] Specifically, the Filebeat component is connected to the Zookeeper node. While the Filebeat component collects local logs, it can calculate the current rate of log collection, and synchronize the rate of these log collection and the status information of the Filebeat component to the Zookeeper node, and update it once within a preset time. The embodiment of the present invention does not limit the preset time, for example, it can be 30s, 1 minute, 3 minutes, etc.

[0049] Then subscribe to the Zookeeper node through the background Filebeat background monitoring manager. The Zookeeper node will push the running status information of the Filebeat component to the Filebeat background monitoring manager to monitor the running status of the Filebeat component. When log collection fails, the root cause of the problem can be quickly located.

[0050] Reference Figure 2 , shows a structural diagram of a Filebeat component provided in an embodiment of the present invention. As can be seen from the figure, after collecting logs from the local, the Filebeat component will cache the collected logs in the Kafka component. While collecting logs, it will also calculate the rate of collecting logs, and synchronize the rate of collecting logs and the status information of the Filebeat component to the Zookeeper node in real time.

[0051] In the embodiment of the present invention, based on the publish-subscribe technology of zookeepr, monitoring is added to the link of log collection. Compared with the manual execution processing method, it not only saves labor costs and shortens the troubleshooting time cycle, but also can quickly locate the problems in the log collection process.

[0052] At the same time, while solving the problem of log collection failure, it can reduce the occupancy rate of server resources, thereby improving the efficiency of log collection.

[0053] Reference Figure 1 The monitoring system also includes an Ec_gostash component; the Ec_gostash component is used to send the running status information of the Ec_gostash component to the Zookeeper node; wherein the running status information of the Ec_gostash component includes the rate at which the Ec_gostash component processes logs and the status information of the Filebeat component; the Zookeeper node is also used to push the running status information of the Ec_gostash component to the background monitoring management program to monitor the running status of the Ec_gostash component.

[0054] Among them, the Logstash component is a tool for collecting, analyzing, and filtering logs. As the log preprocessing and cleaning center, the Logstash component performs simple processing such as filtering and parsing on logs. It supports almost any type of log, including system logs, error logs, and custom application logs. It can receive logs from many sources, including syslog, messaging (such as RabbitMQ), and JMX, and it can output data in a variety of ways, including email, websockets, and Elasticsearch.

[0055] The Ec_gostash component in the embodiment of the present invention is a component developed based on golang, which is used to replace the Logstash component. The Ec_gostash component occupies smaller resources to process more messages. Under the same resource conditions, the efficiency of the Ec_gostash component in processing data is much higher than that of the Logstash component.

[0056] Specifically, the Ec_gostash component is connected to the Zookeeper node. When the Ec_gostash component processes the logs cached in the Kafka queue, it calculates the current log processing rate of the Ec_gostash component and synchronizes the rate information and the status information of the Ec_gostash component to the Zookeeper node.

[0057] Then subscribe to the Zookeeper node through the background Ec_gostash background monitoring management program. The Zookeeper node will push the running status information of the Ec_gostash component to the Ec_gostash background monitoring management program to monitor the running status of the Ec_gostash component. When log collection fails, the root cause of the problem can be quickly located.

[0058] While the Ec_gostash component is processing data, it will calculate the processing rate within 1 minute, 5 minutes, and 15 minutes, and synchronize this information to the zookeeper node. The background Ec_gostash monitoring program subscribes to the zookeeper node to obtain the latest running status of the Ec_gostash component to monitor the running status of the Ec_gostash component.

[0059] In the embodiment of the present invention, based on the publish-subscribe technology of zookeepr, the running status of the Filebeat component and the Ec_gostash component in the log collection process can be mastered, which greatly reduces the time cost of troubleshooting and management. Compared with the manual execution processing method, it not only saves labor costs and shortens the troubleshooting time cycle, but also can quickly locate the problems in the log collection process.

[0060] At the same time, while solving the problem of log collection failure, it can reduce the occupancy rate of server resources, thereby improving the efficiency of log collection.

[0061] In one embodiment of the present invention, the monitoring system also includes a Kafka component and an Elasticsearch component; the Kafka component is used to classify and cache the logs collected by the Filebeat component; the Ec_gostash component is also used to filter the logs cached in the Kafka component and write the filtered logs to the Elasticsearch component for storage.

[0062] Specifically, after collecting logs from the local machine, the Filebeat component will cache the collected logs in the Kafka component. The Ec_gostash component will analyze, filter, and process (consume) the logs cached by the Kafka component, and write the analyzed and filtered logs to the Elasticsearch component for storage. After the Ec_gostash component consumes the logs cached in the Kafka component, it will submit the offset to Kafka.

[0063] In one embodiment of the present invention, the Ec_gostash component includes a Filter module; the Ec_gostash component is also used for a customized field field, the field field maps the corresponding field in the log cached in the Kafka component, and the field field is input into the Filter module to filter the log corresponding to the field field.

[0064] Among them, the existing Ec_gostash components generally include three modules: input, Filter, and output. The filter library of the filter module is rich and diverse, which is used to parse and convert data in real time. An Ec_gostash component can have multiple filters. Commonly used filters include grok, mutate, drop, geoip, etc.

[0065] Specifically, after the Ec_gostash component obtains the logs cached in the Kafka component, it maps the corresponding fields in the logs according to the custom field fields, and then inputs these field fields into the Filter module to filter the logs corresponding to the field fields. Finally, the filtered logs are written to the Elasticsearch component and the offset of the Kafka component is submitted.

[0066] Reference Figure 3, shows a structural schematic diagram of an Ec_gostash component provided in an embodiment of the present invention. As can be seen from the figure, the Ec_gostash component obtains the field field corresponding to the log from the message queue in the Kafka component to filter the log by the field field. When the Ec_gostash component filters the log cached by the Kafka component, it also calculates the rate at which the Ec_gostash component processes the log, and synchronizes the rate at which the Ec_gostash component processes the log and the status information of the Ec_gostash component to the Zookeeper node in real time.

[0067] In one embodiment of the present invention, the Ec_gostash component is further used to record the filtered diaries and the reasons for filtering them using the Filter module, and write the filtered diaries and the reasons for filtering them locally.

[0068] Specifically, after the Filter module of the Ec_gostash component filters the logs, it can record the filtered logs and the reasons for filtering, and write the filtered logs and the reasons for filtering to the local computer, so as to facilitate the later investigation of the reasons why the logs are not correctly written to the elasticsearch component.

[0069] Reference Figure 4 , showing a structural schematic diagram of a Filter module provided in an embodiment of the present invention. When the Ec_gostash component processes the logs cached in the Kafka component, the field field corresponding to the log is input into the Filter module, thereby filtering the logs corresponding to the field field, and writing the successfully filtered logs to the Elasticsearch component. At the same time, the filtered logs and their filtering reasons are recorded, and the filtered logs and filtering reasons are written locally, which is convenient for later investigation of the reasons why the logs are not correctly written to elasticsearch.

[0070] In the embodiment of the present invention, based on the publish-subscribe technology of zookeepr, the running status of the Filebeat component and the Ec_gostash component in the log collection process can be mastered, which greatly reduces the time cost of troubleshooting and management. Compared with the manual execution processing method, it not only saves labor costs and shortens the troubleshooting time cycle, but also can quickly locate the problems in the log collection process.

[0071] At the same time, while solving the problem of log collection failure, it can reduce the occupancy rate of server resources, thereby improving the efficiency of log collection.

[0072] Reference Figure 5 , shows a flow chart of the steps of a monitoring method provided in an embodiment of the present invention, and the specific steps include:

[0073] An embodiment of the present invention discloses a monitoring method, comprising:

[0074] Step 501: Sending the running status information of the Filebeat component to the Zookeeper node through the Filebeat component; wherein the running status information of the Filebeat component includes the rate at which the Filebeat component collects logs from the local and the status information of the Filebeat component;

[0075] Step 502: Push the running status information of the Filebeat component to the background monitoring management program through the Zookeeper node to monitor the running status of the Filebeat component.

[0076] Optionally, it also includes:

[0077] Sending the running status information of the Ec_gostash component to the Zookeeper node through the Ec_gostash component; wherein the running status information of the Ec_gostash component includes the rate at which the Ec_gostash component processes logs and the status information of the Filebeat component;

[0078] The running status information of the Ec_gostash component is pushed to the background monitoring management program through the Zookeeper node to monitor the running status of the Ec_gostash component.

[0079] Optionally, it also includes:

[0080] Classify and cache the logs collected by the Filebeat component through the Kafka component;

[0081] The logs cached in the Kafka component are filtered by the Ec_gostash component, and the filtered logs are written into the Elasticsearch component for storage.

[0082] Optionally, the Ec_gostash component includes a Filter module, and the filtering process of the logs cached in the Kafka component by the Ec_gostash component includes:

[0083] Through the field field customized by the Ec_gostash component, the field field maps the corresponding field in the log cached in the Kafka component, and the field field is input into the Filter module to filter the log corresponding to the field field.

[0084] Optionally, it also includes:

[0085] The Ec_gostash component uses the Filter module to record the filtered diaries and the reasons for filtering, and writes the filtered diaries and the reasons for filtering to the local computer.

[0086] As for the method embodiment, since it is basically similar to the system embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0087] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0088] Memory 603, used for storing computer programs;

[0089] The processor 601 is used to implement the monitoring method described in the above embodiment when executing the program stored in the memory 603.

[0090] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0091] The communication interface is used for communication between the above terminal and other devices.

[0092] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0093] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0094] like Figure 7 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 701 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the monitoring method described in the above embodiment.

[0095] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the monitoring method described in the above embodiment.

[0096] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0097] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0098] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A monitoring system, characterized in that: Including Filebeat components, Zookeeper nodes and background monitoring management programs; among them, The Filebeat component is used to send the running status information of the Filebeat component to the Zookeeper node; wherein the running status information of the Filebeat component includes the rate at which the Filebeat component collects logs from the local and the status information of the Filebeat component; The Zookeeper node is used to push the running status information of the Filebeat component to the background monitoring management program to monitor the running status of the Filebeat component; The background monitoring management program is used to subscribe to the Zookeeper node, and the Zookeeper node pushes the running status information of the Filebeat component to the Filebeat background monitoring management program to monitor the running status of the Filebeat component.

2. The system according to claim 1, characterized in that Also includes Ec_gostash component; The Ec_gostash component is used to send the running status information of the Ec_gostash component to the Zookeeper node; wherein the running status information of the Ec_gostash component includes the rate at which the Ec_gostash component processes logs and the status information of the Filebeat component; The Zookeeper node is also used to push the running status information of the Ec_gostash component to the background monitoring management program to monitor the running status of the Ec_gostash component.

3. The system according to claim 2, characterized in that It also includes Kafka components and Elasticsearch components; The Kafka component is used to classify and cache the logs collected by the Filebeat component; The Ec_gostash component is also used to filter the logs cached in the Kafka component and write the filtered logs into the Elasticsearch component for storage.

4. The system according to claim 3, characterized in that The Ec_gostash component includes a Filter module; The Ec_gostash component is also used for a custom field field, the field field maps the corresponding field in the log cached in the Kafka component, and the field field is input into the Filter module to filter the log corresponding to the field field.

5. The system according to any one of claims 4, characterized in that: The Ec_gostash component is further used to record the filtered diaries and the filtering reasons thereof by using the Filter module, and write the filtered diaries and the filtering reasons into a local computer.

6. A monitoring method, characterized in that: include: Sending the running status information of the Filebeat component to the Zookeeper node through the Filebeat component; wherein the running status information of the Filebeat component includes the rate at which the Filebeat component collects logs from the local and the status information of the Filebeat component; Push the running status information of the Filebeat component to the background monitoring management program through the Zookeeper node to monitor the running status of the Filebeat component; The Zookeeper node is subscribed to through the background monitoring management program, and the Zookeeper node pushes the running status information of the Filebeat component to the Filebeat background monitoring management program to monitor the running status of the Filebeat component.

7. The method according to claim 6, characterized in that Also includes: Sending the running status information of the Ec_gostash component to the Zookeeper node through the Ec_gostash component; wherein the running status information of the Ec_gostash component includes the rate at which the Ec_gostash component processes logs and the status information of the Filebeat component; The running status information of the Ec_gostash component is pushed to the background monitoring management program through the Zookeeper node to monitor the running status of the Ec_gostash component.

8. The method according to claim 7, characterized in that Also includes: Classify and cache the logs collected by the Filebeat component through the Kafka component; The logs cached in the Kafka component are filtered by the Ec_gostash component, and the filtered logs are written into the Elasticsearch component for storage.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 6 to 8 when executing the program stored in the memory.

10. One or more computer-readable media having instructions stored thereon, which when executed by one or more processors cause the processors to perform the method according to any one of claims 6 to 8.

Images