Operation and maintenance monitoring system and method

Abstract

The invention relates to an operation and maintenance monitoring system and method. The operation and maintenance monitoring system comprises a client and a server. The client comprises a data acquisition module and a data output module, and the data acquisition module of the client comprises an agent service, an executable code and an interface; the server comprises a central service module, a storage module, a data collection module, an alarm control module and a web module; the data collection module comprises an acquisition module and a pushing module, and the data collection module can acquire data from a plurality of data sources; the acquisition module can capture data A returned by the interface of the client and data B returned by the data output module; and the pushing module is used for collecting data C provided by the agent service and data D provided by the acquisition module. The data collection mode can reduce the burden of the data collection system while ensuring the real-time performance of the data, and the operation and maintenance monitoring process is efficient, flexible and convenient.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to an operation and maintenance monitoring system and method. Background Art

[0002] In the field of operation and maintenance, the operation and maintenance monitoring system is the core system of operation and maintenance personnel and one of their daily tasks. In the past, operation and maintenance work was usually done manually. Whenever the server scale increased, the manpower input would also increase proportionally. Later, it evolved into a script mode, where one person could monitor the basic indicators of thousands of servers. The script mode evolved into a platform mode, which can be monitored on the web page or other third-party apps, and can perform some daily operation and maintenance, scheduled tasks, alarm self-healing, etc. However, there are problems with this mode: 1. There is no flexibility, and each monitoring requirement requires a large number of scripts; 2. The learning cost is high, and personnel outside the operation and maintenance positions usually need a certain amount of time to learn before they can use it; 3. Alarm coordination and alarm classification are difficult. For example, through agent+server, a large amount of calculations are performed through agents and scripts, and the result data is sent to the server, which stores the data in the database and then analyzes it. However, the use of agent+server not only occupies a large amount of client resources, but also has a data storage bottleneck. When it grows on a large scale, the database load is high and the cost of expanding the server is high; in addition, the configuration is complex, the learning curve is steep, and it requires certain professional skills, which makes it difficult for non-professionals to get started. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide an efficient and accurate operation and maintenance monitoring system and method.

[0004] An operation and maintenance monitoring system of the present invention comprises a client and a server; the client comprises a data acquisition module and a data output module, and the data acquisition module of the client comprises an agent service, an executable code and an interface; The server includes a central service module, a storage module, a data collection module, an alarm control module and a web module; The data collection module includes a collection module and a push module. The data collection module can obtain data from multiple data sources; the collection module can capture data A returned by the client interface and data B returned by the data output module; the push module includes collecting data C provided by the agent service and data D provided by the collection module; The storage module uses the m3db time series database as the storage component of the operation and maintenance monitoring system, and the storage module supports horizontal expansion; The central service module includes a user management module, a team management module and a business group management module; the business group management module includes an alarm service, and the alarm service includes an alarm rule.

[0005] Furthermore, the alarm rules of the operation and maintenance monitoring system described in the present application include indicator values ​​and / or comparison symbols and / or alarm channels and / or alarm recipients or groups and / or alarm time periods; The indicator value, including null or a number; The comparison symbols include one or more combinations of <, >, <=, >=, =, !=, in, or, not in, and; The alarm channel includes one or more combinations of phone calls, text messages, emails, and webhooks; The alarm recipient or group includes a single person, multiple people, a single group, or a combination of one or more of multiple groups; The alarm period includes sending only a designated channel within a designated time.

[0006] Furthermore, the executable code of the operation and maintenance monitoring system described in the present application can collect data through shell or python or go script and output personalized data in a specified format. The agent service includes collecting the basic monitoring data and the personalized data of the client, and pushing the basic monitoring data and the personalized data to the server; the basic monitoring data includes cpu, memory, hard disk, network, IO and load conditions, and the data output module includes various exporters of prometheus.

[0007] Furthermore, the data source of the operation and maintenance monitoring system described in the present application includes data stored in prometheus and / or data stored in elasticsearch and / or data stored in loki and / or data stored in TDengine and / or data stored in tsdb and / or data stored in mysql and / or data stored in redis.

[0008] Furthermore, the operation and maintenance monitoring system described in the present application also includes a user end, and the user end includes an operation and maintenance end and a research and development end. The operation and maintenance end has a modification function, and the modification content includes alarm rules, monitoring collection items, authority division and user management; the research and development end only has non-administrator functions, and the non-administrator functions are alarm claiming and / or modifying personal information and / or monitoring data viewing.

[0009] Furthermore, the user management module of the operation and maintenance monitoring system described in the present application includes user information and user authority settings of the user, wherein the user information includes display name, role, email address, mobile phone number, and webhook; each user is set with at least one role; The team management module includes team setting, and the specific process of the team setting includes dividing the team according to departments, and the team includes at least one user.

[0010] Furthermore, the business group management module of the operation and maintenance monitoring system described in the present application also includes monitoring services, and the monitoring services include alarm rules, alarm time and monitoring objects. The self-healing script is grouped according to the services and the service affiliation is set.

[0011] Furthermore, the Web services provided by the web module of the operation and maintenance monitoring system described in the present application include managing the backend, frontend, nginx of the background page, managing the operation and maintenance monitoring system, setting alarms, and monitoring the dashboard display.

[0012] On the other hand, the present application also proposes an operation and maintenance monitoring method based on the operation and maintenance monitoring system described in any one of the above items, which specifically includes the following steps: 1) Set the content of the central service module through the web module and store it in the storage module. The setting content of the central service module includes user information, user authority setting, team setting, alarm service, and monitoring service; 2) The data collection module completes the data collection process and aggregates the collected data and sends it to the storage module. The data collection process specifically collects data C provided by the agent service and data D provided by the collection module through the push module. The collection module collects data D by grabbing data A returned by the client interface and data B returned by the data output module; 3) The alarm control module determines and generates an alarm service based on the content in the storage module, and the alarm service is pushed to the cache service and the database service; 4) The alarm control module completes the alarm business according to the alarm rules of the database service.

[0013] Furthermore, the cache service of the operation and maintenance monitoring system of the present invention is used to store the alarm services currently to be sent, and the database service is used to store historical alarm service records.

[0014] Compared with the prior art, the present invention has the following beneficial technical effects: The present invention solves the problems existing in the background technology through a new data storage and push-pull combined data collection mode. The data collection mode combining push and pull is realized through a data collection module, which can reduce the burden of the data collection system while ensuring the real-time performance of the data, and improve the efficiency and accuracy of data collection.

[0015] On the one hand, for operation and maintenance personnel, deployment is faster, the impact on clients is smaller, and configuration is simpler.

[0016] On the other hand, for employees in other companies, there is no need to communicate directly with the operation and maintenance staff for small issues. They can obtain the content they need through the web page, such as adding alarms, claiming alarms, viewing data dashboards, and other daily repetitive work.

[0017] The operation and maintenance monitoring system and method described in this application not only has good alarm coordination, but also has simple and accurate alarm classification. In addition, it also has the advantages of data collection, visualization, high performance, easy expansion, flexible configuration, and can support real-time data query and historical data statistical analysis; support multiple alarm trigger conditions and alarm channels. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the operation and maintenance monitoring system of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with specific embodiments. Specific implementation method one:

[0020] An operation and maintenance monitoring system, comprising a client and a server; the client comprises a data acquisition module and a data output module, and the data acquisition module of the client comprises an agent service, an executable code and an interface; The server includes a central service module, a storage module, a data collection module, an alarm control module and a web module; The data collection module includes a collection module and a push module. The data collection module can obtain data from multiple data sources; the collection module can capture data A returned by the client interface and data B returned by the data output module; the push module includes collecting data C provided by the agent service and data D provided by the collection module; The storage module uses the m3db time series database as the storage component of the operation and maintenance monitoring system, and the storage module supports horizontal expansion; The central service module includes a user management module, a team management module and a business group management module; the business group management module includes an alarm service, and the alarm service includes an alarm rule.

[0021] In some other embodiments, the alarm rule includes an indicator value and / or a comparison symbol and / or an alarm channel and / or an alarm recipient or group and / or an alarm period; The indicator value, including null or a number; The comparison symbols include one or more combinations of <, >, <=, >=, =, !=, in, or, not in, and; The alarm channel includes one or more combinations of phone calls, text messages, emails, and webhooks; The alarm recipient or group includes a single person, multiple people, a single group, or a combination of one or more of multiple groups; The alarm period includes sending only a designated channel within a designated time.

[0022] In some other embodiments, the executable code can collect data through shell or python or go script and output personalized data in a specified format. The agent service includes collecting basic monitoring data and personalized data of the client, and pushing the basic monitoring data and personalized data to the server; the basic monitoring data includes cpu, memory, hard disk, network, IO and load conditions, and the data output module includes various exporters of prometheus.

[0023] In some other embodiments, the data source includes data stored in prometheus and / or data stored in elasticsearch and / or data stored in loki and / or data stored in TDengine and / or data stored in tsdb and / or data stored in mysql and / or data stored in redis.

[0024] In some other embodiments, the operation and maintenance monitoring system also includes a user end, which includes an operation and maintenance end and a research and development end. The operation and maintenance end has a modification function, and the modification content includes alarm rules, monitoring collection items, authority division and user management; the research and development end only has non-administrator functions, and the non-administrator functions are alarm claiming and / or modifying personal information and / or monitoring data viewing.

[0025] In some other implementations, the user management module includes user information and user permission settings of the user, the user information includes display name, role, email address, mobile phone number, webhook; each user sets at least one role; The team management module includes team setting, and the specific process of the team setting includes dividing the team according to departments, and the team includes at least one user.

[0026] In some other implementations, the service group management module further includes monitoring services, and the monitoring services include alarm rules, alarm time, and monitoring objects. The self-healing script is grouped according to the services and the service affiliation is set.

[0027] In some other implementations, the Web services provided by the web module include managing the backend, frontend, and nginx of the background page, managing the operation and maintenance monitoring system, setting alarms, and monitoring dashboard display. Specific implementation method 2:

[0028] An operation and maintenance monitoring method based on the operation and maintenance monitoring system described in any of the above embodiments specifically comprises the following steps: 1) Set the content of the central service module through the web module and store it in the storage module. The setting content of the central service module includes user information, user authority setting, team setting, alarm service, and monitoring service; 2) The data collection module completes the data collection process and aggregates the collected data and sends it to the storage module. The data collection process specifically collects data C provided by the agent service and data D provided by the collection module through the push module. The collection module collects data D by grabbing data A returned by the client interface and data B returned by the data output module; 3) The alarm control module determines and generates an alarm service based on the content in the storage module, and the alarm service is pushed to the cache service and the database service; 4) The alarm control module completes the alarm business according to the alarm rules of the database service.

[0029] In other implementations, the cache service is used to store the alarm services currently to be sent, and the database service is used to store historical alarm service records. Embodiment 1:

[0030] An operation and maintenance monitoring system, such as Figure 1 As shown, it includes a client, a server and a user; the client includes a data acquisition module and a data output module, and the data acquisition module of the client includes an agent service, executable code and an interface; the server includes a central service module, a storage module, a data collection module, an alarm control and a web module. The user includes an operation and maintenance end and a research and development end. The operation and maintenance end has a modification function, and the modification content includes alarm rules, monitoring and collection items, authority division and user management; the research and development end only has non-administrator functions, and the non-administrator functions are alarm claiming and / or modifying personal information and / or monitoring data viewing.

[0031] The executable code can collect data through shell or python or go script and output personalized data in a specified format. The agent service includes collecting the basic monitoring data and the personalized data of the client, and pushing the basic monitoring data and the personalized data to the server; the basic monitoring data includes cpu, memory, hard disk, network, IO and load conditions, and the data output module includes various exporters of prometheus.

[0032] The data collection module includes an acquisition module and a push module. The data collection module can obtain data from multiple data sources; the acquisition module can capture data A returned by the client interface and data B returned by the data output module; the push module includes collecting data C provided by the agent service and data D provided by the acquisition module.

[0033] The data source includes data stored in prometheus and / or data stored in elasticsearch and / or data stored in loki and / or data stored in TDengine and / or data stored in tsdb and / or data stored in mysql and / or data stored in redis.

[0034] The storage module uses the m3db time series database as the storage component of the operation and maintenance monitoring system, and the storage module supports horizontal expansion; The central service module includes a user management module, a team management module and a business group management module; the business group management module includes an alarm service and a monitoring service, and the alarm service includes an alarm rule. The monitoring service includes an alarm rule, an alarm time and a monitoring object. The self-healing script is grouped according to the service and the service affiliation is set.

[0035] The user management module includes user information and user authority settings of the user, wherein the user information includes display name, role, email address, mobile phone number, and webhook; each user is set with at least one role; the team management module includes team settings, wherein the specific process of team settings includes dividing the team according to departments, and the team includes at least one user.

[0036] The alarm rules include indicator values ​​and / or comparison symbols and / or alarm channels and / or alarm recipients or groups and / or alarm time periods; The indicator value, including null or a number; The comparison symbols include one or more combinations of <, >, <=, >=, =, !=, in, or, not in, and; The alarm channel includes one or more combinations of phone calls, text messages, emails, and webhooks; The alarm recipient or group includes a single person, multiple people, a single group, or a combination of one or more of multiple groups; The alarm period includes sending only a designated channel within a designated time.

[0037] The web services provided by the web module include managing the backend, frontend, nginx of the background page, managing the operation and maintenance monitoring system, setting alarms, and monitoring the dashboard display.

[0038] In this embodiment 1, the deployment content includes: agent port 7000, web-server port 8080, collect-server port 8081, data-server port 8082, center-server port 8083, alert-server port 8084, m3db port 9000, mysql and redis are deployed by themselves, and the client also includes prometheus exporter.

[0039] Single-node deployment: Each service can run directly and access each other through the http protocol. --config specifies the service configuration file. The default is the config file with the same name as the service in the current directory. It cannot run without a configuration file.

[0040] Multi-node deployment: Each service supports horizontal expansion to improve processing performance. After deployment, you only need to add multiple addresses to the config file of the dependent service.

[0041] In this embodiment 1, the specific process of adding the alarm rule includes: logging into the web page, entering the alarm management, and then adding the alarm rule: Fill in the form: rule name, additional tags, remarks, data source, alarm condition, alarm level, execution frequency, duration, alarm person or group, sending time, channel configuration, repetition interval, recovery notification, callback address.

[0042] Indicator query: The overall indicators can be viewed on the web page dashboard, specifically in the monitoring dashboard.

[0043] Detailed indicators can be found on the web page Time Series Indicators, where you can instantly query indicators, tags, rules, etc.

[0044] Specific personnel configuration in this embodiment 1: Personnel are managed from three dimensions: team group, business group, and ordinary user. A user can be in multiple groups and teams at the same time. The team is the department, and the business group is the project member. The group here can be set as the notification object by the alarm rule, and the personnel authority is set at the group level or ordinary user. The authority division includes dashboard, alarm setting, alarm viewing, indicator setting, user setting, and system setting. Embodiment 2:

[0045] Based on the usage scenarios of operation and maintenance personnel: A company has hundreds of servers, which belong to multiple business groups respectively. After the machines are batch-agented, they enter the operation and maintenance monitoring system described in this application for management. It has infrastructure management, which is equivalent to a simple CMDB system with hardware information. Some command operations can be performed in batches. There are some special monitoring requirements, such as monitoring some personalized systems. You only need to write a python script to obtain system interface data and push the data to the local agent. The agent will report the data to the monitoring system together and set the monitoring threshold in the monitoring system to realize the monitoring of personalized systems. Moreover, the monitoring system is compatible with the monitoring plug-in of prometheus. The data of plug-ins such as mysql_exporter, redis_exporter, and kafka_exporter can be captured, and subsequent operations such as query, alarm, and dashboard in the system are performed as the client of the operation and maintenance monitoring system described in this application. Embodiment 3:

[0046] Based on the usage scenarios of non-operation and maintenance personnel: The company has hundreds of servers, which belong to multiple business groups. The servers are divided into business groups. Business group members can only view the server monitoring data within their group. Server alarm information will only be sent to the alarm channel of the corresponding group member, realizing authority division. After a member of the group claims the corresponding alarm, other members will receive the alarm claim notification, so that they know that someone is already handling the alarm, which improves collaboration and reduces communication costs.

Claims

1. An operation and maintenance monitoring system, characterized in that: It includes a client and a server; the client includes a data acquisition module and a data output module, and the data acquisition module of the client includes an agent service, an executable code and an interface; The server includes a central service module, a storage module, a data collection module, an alarm control module and a web module; The data collection module includes a collection module and a push module. The data collection module can obtain data from multiple data sources; the collection module can capture data A returned by the client interface and data B returned by the data output module; the push module includes collecting data C provided by the agent service and data D provided by the collection module; The storage module uses the m3db time series database as the storage component of the operation and maintenance monitoring system, and the storage module supports horizontal expansion; The central service module includes a user management module, a team management module and a business group management module; the business group management module includes an alarm service, and the alarm service includes an alarm rule.

2. The operation and maintenance monitoring system according to claim 1, characterized in that: The alarm rules include indicator values ​​and / or comparison symbols and / or alarm channels and / or alarm recipients or groups and / or alarm time periods; The indicator value, including null or a number; The comparison symbols include one or more combinations of <, >, <=, >=, =, !=, in, or, not in, and; The alarm channel includes one or more combinations of phone calls, text messages, emails, and webhooks; The alarm recipient or group includes a single person, multiple people, a single group, or a combination of one or more of multiple groups; The alarm period includes sending only a designated channel within a designated time.

3. The operation and maintenance monitoring system according to claim 2, characterized in that: The executable code can collect data through shell or python or go script and output personalized data in a specified format. The agent service includes collecting the basic monitoring data and the personalized data of the client, and pushing the basic monitoring data and the personalized data to the server; the basic monitoring data includes cpu, memory, hard disk, network, IO and load conditions, and the data output module includes various exporters of prometheus.

4. The operation and maintenance monitoring system according to claim 3, characterized in that: The data source includes data stored in prometheus and / or data stored in elasticsearch and / or data stored in loki and / or data stored in TDengine and / or data stored in tsdb and / or data stored in mysql and / or data stored in redis.

5. The operation and maintenance monitoring system according to claim 4, characterized in that: The operation and maintenance monitoring system also includes a user end, which includes an operation and maintenance end and a research and development end. The operation and maintenance end has a modification function, and the modification content includes alarm rules, monitoring collection items, authority division and user management; the research and development end only has non-administrator functions, and the non-administrator functions are alarm claiming and / or modifying personal information and / or monitoring data viewing.

6. The operation and maintenance monitoring system according to claim 5, characterized in that: The user management module includes the user information and user authority settings of the user, and the user information includes display name, role, email, mobile phone number, webhook; each user is set with at least one role; The team management module includes team setting, and the specific process of the team setting includes dividing the team according to departments, and the team includes at least one user.

7. The operation and maintenance monitoring system according to claim 6, characterized in that: The service group management module also includes monitoring services, which include alarm rules, alarm time and monitoring objects. The self-healing script is grouped according to the services and the service affiliation is set.

8. The operation and maintenance monitoring system according to claim 7, characterized in that: The web services provided by the web module include managing the backend, frontend, nginx of the background page, managing the operation and maintenance monitoring system, setting alarms, and monitoring the dashboard display.

9. An operation and maintenance monitoring method, characterized in that: The operation and maintenance monitoring method is based on the operation and maintenance monitoring system according to any one of claims 1 to 8, and specifically comprises the following steps: 1) Set the content of the central service module through the web module and store it in the storage module. The setting content of the central service module includes user information, user authority setting, team setting, alarm service, and monitoring service; 2) The data collection module completes the data collection process and aggregates the collected data and sends it to the storage module. The data collection process specifically collects data C provided by the agent service and data D provided by the collection module through the push module. The collection module collects data D by grabbing data A returned by the client interface and data B returned by the data output module; 3) The alarm control module determines and generates an alarm service based on the content in the storage module, and the alarm service is pushed to the cache service and the database service; 4) The alarm control module completes the alarm business according to the alarm rules of the database service.

10. The operation and maintenance monitoring system according to claim 9, characterized in that: The cache service is used to store the alarm services currently to be sent, and the database service is used to store historical alarm service records.

Images