A MongoDB database monitoring method based on Kubernetes container technology

Abstract

The present invention provides a MongoDB database monitoring method based on Kubernetes container technology, which belongs to the technical field of virtual machine indicator monitoring in a Kubernetes environment. The present invention utilizes various plug-ins of Telegraf deployed in a Pod container to complete filtering and storage redirection of collected indicators by formulating configuration files. Business monitoring indicators, application monitoring indicators, and network monitoring indicators are collected into the same logical table. Data is marked and classified using global tags. Data is added with identifiers and stored in Influxdb deployed in the Pod container. The collection and classified storage of various monitoring indicator data can be completed without involving modification of the Telegraf plug-in, so that only valid data is collected and summarized, storage costs are saved, and storage and query efficiency are improved. The technical requirements for the use of the solution are reduced, and it is easy to use.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual machine indicator monitoring in a Kubernetes environment, and in particular to a MongoDB database monitoring method based on Kubernetes container technology. Background Art

[0002] Kubernetes is the most popular container cluster management platform that has become the de facto standard. It provides a series of complete functions for containerized applications, such as clustered deployment and operation, automatic resource scheduling, and dynamic horizontal scaling. Although the Kubernetes platform itself has realized real-time monitoring of application status, the monitoring indicators and methods are still relatively basic and difficult to meet various complex and personalized monitoring and management needs. Therefore, we need to add an independent monitoring system based on Kubernetes that does not affect the architecture and deployment of existing application clusters, and is comprehensive, easy to deploy, easy to monitor and analyze, and can issue timely alarms.

[0003] Telegraf is an agent written in Go that collects and reports metrics and data. It collects statistics about systems and services and writes them to the InfluxDB database. Telegraf has a small memory footprint, and developers can easily add extensions to support other services through the plugin system.

[0004] The solutions provided by many open source monitoring software can solve monitoring problems at some levels, but they cannot provide an integrated resource monitoring solution for open source cloud platforms from a holistic and multi-faceted perspective.

[0005] Most existing Telegraf collection applications use the method of modifying the plug-in source code to achieve data filtering and aggregation, which requires a high level of technical skills from the users, and the source code modification and verification process is complex and inefficient. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a MongoDB database monitoring method based on Kubernetes container technology, which performs multi-dimensional and comprehensive real-time monitoring of resource information such as the service status of the MongoDB database and the status of the virtual machine.

[0007] The technical solution of the present invention is:

[0008] A MongoDB database monitoring method based on Kubernetes container technology uses container technology to deploy Telegraf applications and MongoDB applications in the same container. The aggregation of MongoDB monitoring indicators and virtual machine monitoring indicators is completed by modifying the Telegraf configuration file. The dimensions of Telegraf are set by reading the virtual machine environment information. The monitoring indicators are collected by mounting the virtual machine system directory through Docker.

[0009] Furthermore,

[0010] Utilize various Telegraf plug-ins deployed in the Pod container and formulate configuration files to complete the filtering and storage redirection of collected indicators;

[0011] Collect business monitoring indicators, application monitoring indicators, and network monitoring indicators into the same logical table;

[0012] Use global tags to mark and classify data, add identifiers to the data and store them in Influxdb deployed in the Pod container.

[0013] Going further,

[0014] MongoDB is deployed in the Docker container of Kubernetes. The Influxdb database Pod is deployed in the Kubernetes environment. A domain name for storage is provided for the virtual machine. The CMS service Pod that provides monitoring data query is deployed.

[0015] Deploy Telegraf on the virtual machine where MongoDB is located.

[0016] Monitoring data collection includes data collection module and data storage module. The data collection module uses the monitoring tool Telegraf and the plug-in mechanism to collect various resource monitoring information. The data storage module uses the Influxdb time series database.

[0017] Furthermore,

[0018] Deploy the MongoDB application using Docker in the virtual machine created in the Kubernetes environment through Ansible;

[0019] Use the script to obtain the ID of the deployed virtual machine as the global tag resource_id of the collection indicator, read the network card information of the virtual machine, and take the IP as the global tag user_ip of the collection indicator; start the container through the Docker command.

[0020] When the container is started, it is necessary to mount the disk directory for data storage and the component directories / proc: / hostfs / proc and / sys: / hostfs / sys for collecting disk information, and set environment variables to collect disk information.

[0021] By modifying the Telegraf configuration file, set the global tag; take the environment variables resource_id, user_ip, etc. passed in when Docker is started to set [global_tags], so that each piece of monitoring data collected has a global tag, and can be filtered according to needs during query.

[0022] By modifying the Telegraf configuration file, the filtering of the collected indicators is completed; each component in Telegraf collects the full amount of monitoring storage data into its own Influxdb corresponding table, thus generating redundant data; here, the collection attributes of each indicator are selected by configuring fieldpass filtering for each collected indicator; the collection hardware is specified through the component, the interfaces of the net component are used to configure the specified network card for collection, and the mount_points of the disk component are used to specify the collected disk.

[0023] Aggregate monitoring indicators by modifying the Telegraf configuration file; use the processors.rename component, set processors.rename.replace, specify measurement and dest, and redirect the monitoring data collected by the measurement component to the dest table; use field and dest to redirect the collected indicator attributes to the fields in the target table;

[0024] If the measurement field is duplicated in more than one measurement, you need to add namepass under the processors.rename tag to specify the monitoring data source component.

[0025] By configuring outputs.influxdb and specifying database and retention_policy, data isolation and regular clearing can be achieved.

[0026] Use the cpu component to collect the virtual machine CPU idle usage rate usage_idle (use 100 minus usage_idle as the CPU usage rate when querying), and rename the indicator to the node_cpu_ratio attribute of the mongo table;

[0027] Use the mem component to collect the used (usage) and used_percent (usage rate) of the virtual machine memory, and rename the indicators to node_used_memory and node_used_memory_ratio of the mongo table;

[0028] Use the disk component to collect the used (usage rate) of the virtual machine data disk data and rename the indicator to node_used_disk of the mongo table;

[0029] Use the net component to collect bytes_recv (data received) and bytes_send (data sent) of the virtual machine network card eth0, and rename the indicators to node_net_recv and node_net_send of the mongo table;

[0030] Use the mongodb component to collect MongoDB indicators such as connections_current, delete_command_total, insert_command_total, etc., and rename the indicators to the mongo table;

[0031] Data query: After the monitoring indicator data is stored in Influxdb, you only need to query the mongo table when querying.

[0032] The beneficial effects of the present invention are

[0033] It can complete the collection and classified storage of various monitoring indicator data without involving the modification of Telegraf plug-ins, so as to collect and summarize only effective data, save storage costs, improve storage and query efficiency, reduce the technical requirements for the use of the solution, and facilitate use. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a deployment schematic diagram of the present invention;

[0035] Figure 2 It is a schematic diagram of the deployment and implementation of the present invention;

[0036] Figure 3 It is a schematic diagram of data conversion of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] The present invention provides a MongoDB database monitoring method based on Kubernetes, which mainly solves the problem of collecting various monitoring indicators of MongoDB service deployed in Docker in Kubernetes environment without modifying Telegraf source code. It does not rely on other third-party monitoring tools. It accurately collects and stores monitoring indicators in Influxdb database, reducing redundant data.

[0039] MongoDB is deployed in the Docker container of Kubernetes. The Kubernetes environment deploys the Influxdb database Pod, provides a domain name for storage on the virtual machine, and deploys the CMS service Pod that provides monitoring data query. Telegraf is deployed on the virtual machine where MongoDB is located. Monitoring data collection mainly includes data collection module and data storage module. The data collection module uses the monitoring tool Telegraf and uses the plug-in mechanism to collect various resource monitoring information. The data storage module uses the Influxdb time series database.

[0040] The first aspect is to use Ansible to deploy the MongoDB application in the virtual machine that has been created in the Kubernetes environment using Docker. The deployment environment is as follows Figure 1 Shown

[0041] Secondly, the script is used to obtain the ID of the deployed virtual machine as the global tag resource_id of the collected indicators, read the network card information of the virtual machine, and take the IP as the global tag user_ip of the collected indicators. The container is started through the Docker command.

[0042] When the container is started, it is necessary to mount the disk directory for data storage and the component directories / proc: / hostfs / proc and / sys: / hostfs / sys for collecting disk information, and set environment variables to collect disk information.

[0043] The third aspect is to set global tags by modifying the Telegraf configuration file. Set [global_tags] using the environment variables resource_id, user_ip, etc. passed in when Docker is started. This way, each piece of monitoring data collected has a global tag, which can be filtered according to needs when querying.

[0044] Fourthly, the filtering of the collected indicators is completed by modifying the Telegraf configuration file. Each component in Telegraf collects the full amount of monitoring storage data into its own corresponding Influxdb table, thus generating redundant data. Here, the collection attributes of each indicator are selected by configuring fieldpass filtering for each collected indicator. The collection hardware is specified through the component, the interfaces of the net component are used to configure the collection network card, and the mount_points of the disk component are used to specify the collection disk.

[0045] The fifth aspect is to aggregate monitoring indicators by modifying the Telegraf configuration file. Use the processors.rename component, set processors.rename.replace, specify measurement and dest, and redirect the monitoring data collected by the measurement component to the dest table. Through field and dest, redirect the collected indicator attributes to the fields in the target table. If the field of the collected measurement has duplicate items in multiple measurements, for example, both mem and disk exist and need to be collected as used, you need to add namepass under the processors.rename tag to specify the monitoring data source component.

[0046] Sixth, by configuring outputs.influxdb and specifying database and retention_policy, data isolation and regular clearing can be achieved.

[0047] The seventh aspect is the specific implementation plan. Use the cpu component to collect the virtual machine CPU idle usage rate usage_idle (use 100 minus usage_idle as the CPU usage rate when querying), and rename the indicator to the node_cpu_ratio attribute of the mongo table. Use the mem component to collect the used (usage) and used_percent (usage rate) of the virtual machine memory, and rename the indicators to node_used_memory and node_used_memory_ratio of the mongo table. Use the disk component to collect the used (usage rate) of the virtual machine data disk data, and rename the indicator to node_used_disk of the mongo table. Use the net component to collect the bytes_recv (data received) and bytes_send (data sent) of the virtual machine network card eth0, and rename the indicators to node_net_recv and node_net_send of the mongo table. Use the mongodb component to collect MongoDB indicators such as connections_current, delete_command_total, insert_command_total, etc., and rename the indicators to the mongo table. The specific implementation is as follows Figure 2 The processing process is shown in the attached Figure 3 shown.

[0048] The eighth aspect is data query. After the monitoring indicator data is stored in Influxdb, you only need to query the mongo table when querying. You can do similar queries

[0049] Select node_cpu_ratio,node_used_memory,ode_used_memory_ratio,node_net_recv,node_net_send,connections_current,delete_command_total,insert_command_total from mongodb.mongo where resource_id=***and user_ip=***

[0050] Define filed in the Influxdb database as the minimum storage unit, define the collection plug-in corresponding to the measurement to have n fields, the number of collection indicators to be m, and the number of global tags to be k. Then the storage capacity generated by using the conventional Telegraf configuration solution to collect data is

[0051]

[0052] The storage capacity generated by collecting data using the Telegraf configuration scheme of this invention is m+k, and the data storage ratio is

[0053]

[0054] When the number of indicator fields n, the number of collected indicators m, and the number of global tags k increase, the storage ratio becomes smaller. It can be seen that when the number of collected indicators, the number of indicator fields, and the number of global tags increase, this solution can save more storage space.

[0055] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A MongoDB database monitoring method based on Kubernetes container technology, characterized in that: Use container technology to deploy the Telegraf application and the MongoDB application in the same container; Aggregate MongoDB monitoring indicators and virtual machine monitoring indicators by modifying the Telegraf configuration file; Set Telegraf dimensions by reading virtual machine environment information; Collect monitoring indicators by mounting the virtual machine system directory through Docker; Utilize various Telegraf plug-ins deployed in the Pod container and formulate configuration files to complete the filtering and storage redirection of collected indicators; Collect business monitoring indicators, application monitoring indicators, and network monitoring indicators into the same logical table; Use global tags to mark and classify data, add identifiers to the data and store them in Influxdb deployed in the Pod container.

2. The method according to claim 1, characterized in that: MongoDB is deployed in the Docker container of Kubernetes. The Influxdb database Pod is deployed in the Kubernetes environment. A domain name for storage is provided for the virtual machine. The CMS service Pod for monitoring data query is deployed. Deploy Telegraf on the virtual machine where MongoDB is located; Monitoring data collection includes data collection module and data storage module. The data collection module uses the monitoring tool Telegraf and the plug-in mechanism to collect various resource monitoring information. The data storage module uses the Influxdb time series database.

3. The method according to claim 2, characterized in that Deploy the MongoDB application using Docker in the virtual machine created in the Kubernetes environment through Ansible; Get the ID of the deployed virtual machine through the script as the global tag resource_id of the collection indicator, read the network card information of the virtual machine, and take the IP as the global tag user_ip of the collection indicator; start the container through the Docker command; When the container is started, it is necessary to mount the disk directory for data storage and the component directories / proc: / hostfs / proc and / sys: / hostfs / sys for collecting disk information, and set environment variables to collect disk information.

4. The method according to claim 2, characterized in that: By modifying the Telegraf configuration file, set the global tag; take the environment variables resource_id and user_ip passed in when Docker is started to set global_tags, so that each piece of monitoring data collected has a global tag, and can be filtered according to needs during query.

5. The method according to claim 2, characterized in that: By modifying the Telegraf configuration file, the filtering of the collected indicators is completed; each component in Telegraf collects the full amount of monitoring storage data into its own Influxdb corresponding table, thus generating redundant data; by configuring fieldpass filtering for each collected indicator, the collection attributes of each indicator are selected; the collection hardware is specified through the component, the interfaces of the net component are used to configure the specified network card for collection, and the mount_points of the disk component are used to specify the collected disk.

6. The method according to claim 2, characterized in that Aggregate monitoring indicators by modifying the Telegraf configuration file; use the processors.rename component, set processors.rename.replace, specify measurement and dest, and redirect the monitoring data collected by the measurement component to the dest table; use field and dest to redirect the collected indicator attributes to the fields in the target table.

7. The method according to claim 6, characterized in that If the measurement field is duplicated in more than one measurement, you need to add namepass under the processors.rename tag to specify the monitoring data source component.

8. The method according to claim 2, characterized in that: By configuring outputs.influxdb and specifying database and retention_policy, data isolation and regular clearing can be achieved.

9. The method according to claim 2, characterized in that: Use the cpu component to collect the virtual machine CPU idle usage rate usage_idle. When querying, subtract usage_idle from 100 as the CPU usage rate. Rename the indicator to the node_cpu_ratio attribute of the mongo table. Use the mem component to collect the virtual machine memory usage used and usage rate used_percent, and rename the indicators to node_used_memory and node_used_memory_ratio in the mongo table; Use the disk component to collect the usage rate of the virtual machine data disk data, and rename the indicator to node_used_disk in the mongo table; Use the net component to collect the data received by the virtual machine network card eth0, bytes_recv, and the data sent by bytes_send, and rename the indicators to node_net_recv and node_net_send in the mongo table; Use the mongodb component to collect MongoDB indicators connections_current, delete_command_total, insert_command_total, and rename the indicators to the mongo table; Data query: After the monitoring indicator data is stored in Influxdb, you only need to query the mongo table when querying.

Images