System and method for monitoring storage capacity of containerized deployment disk

Abstract

The invention discloses a containerized deployment disk storage capacity monitoring system and method, and the method comprises the steps: collecting and storing monitoring data through Prometheus, and achieving a flexible alarm mechanism through Alertmanager; the method comprises the following steps: acquiring storage capacity information of a non-mounting point disk of a host in a manner of developing Exporter, and monitoring middleware service of a host disk; the disk capacity condition is concerned at any time through the visual monitoring board, so that abnormal system service operation caused by insufficient disks is prevented; the efficiency and the reliability of middleware management are improved by monitoring the storage capacity of a non-mounting point disk of the containerized middleware Pod in a Hostpath storage mode.

Description

Technical Field

[0001] The present invention relates to the field of software operation and maintenance, and in particular to a system and method for monitoring the storage capacity of a containerized deployed disk. Background Art

[0002] Docker is an open source containerization platform that uses containerization technology to achieve rapid deployment, portability, and scalability of applications. Docker uses operating system-level virtualization technology to package applications and their dependencies into an independent operating environment called a container, enabling them to run in the same way in different computing environments.

[0003] Kubernetes is an open source container orchestration and management platform for automating the deployment, expansion, and operation of application containers. It provides a rich set of features to help users simplify the management and operation of containerized applications.

[0004] Prometheus is an open source system monitoring and alerting tool originally developed by SoundCloud and now part of CNCF. It collects time series data by scraping indicator data on HTTP endpoints and provides flexible monitoring query and alerting capabilities.

[0005] Grafana is an open source visualization and monitoring tool that is widely used to display time series data. It integrates with data sources such as Prometheus, InfluxDB, and Graphite to create dynamic dashboards and charts.

[0006] The present invention combines Docker, Kubernetes, Prometheus, and Grafana to create a system and method for monitoring disk storage capacity in containerized deployment. Summary of the invention

[0007] The purpose of the present invention is to provide a system and method for monitoring disk storage capacity in a containerized manner in response to the above-mentioned technical problems.

[0008] A system for monitoring disk storage capacity in containerized deployment, including a data indicator collection module, a data indicator calculation module, and a data indicator exposure module; The data indicator collection module collects non-mount point disk capacity numerical indicators used by the containerized deployment middleware; The data indicator calculation module calculates the non-mount point disk capacity numerical indicator used by the containerized deployment middleware; The data indicator exposure module exposes the non-mount point disk capacity numerical indicator used by the containerized deployment middleware in an http manner.

[0009] Furthermore, a system for monitoring disk storage capacity in containerized deployment is provided, wherein the function of the data indicator collection module is to determine whether a directory exists based on the disk directory mounted by the container pod, and if the disk directory exists, a system command is called to collect the disk capacity value returned by the system command.

[0010] Further, in a system for monitoring disk storage capacity in containerized deployment, the data indicator calculation module functions to perform unit calculation and conversion based on the data value returned by the data indicator collection module; The units include KB, MB, and GB.

[0011] Furthermore, in a system for monitoring disk storage capacity in a containerized deployment, the function of the data indicator exposure module is to name the values ​​calculated by the data indicator calculation module and encapsulate them as Prometheus indicators for exposure.

[0012] A method for monitoring disk storage capacity in containerized deployment includes the following sub-steps: S1: Create a Docker image and set up pre-configured non-mount point disk capacity data collection applications and related tools; S2: Use Golang to write an application Exporter that collects non-mount point disk capacity data and generates Prometheus monitoring indicators; S3: Exposes indicators through Http, and the Prometheus client library pushes data to the Prometheus service; S4: Deploy the application for collecting non-mount point disk capacity data to each host that needs to collect non-mount point disk storage capacity data through the K8s Daemonset mode, and check the service operation status; S5: Configure Prometheus to collect non-mount point disk storage capacity indicator data; S6: Configure the Grafana dashboard and write PromQL to display the non-mount point disk storage capacity indicator data; S7: Monitor the non-mounted disk capacity through the containerized middleware non-mounted disk capacity data collection system.

[0013] Furthermore, a method for monitoring disk storage capacity in containerized deployment is provided, wherein the related tools include a configuration file for collecting non-mount point disk capacity data, an initialization script, and necessary dependencies.

[0014] The beneficial effects of the present invention are as follows: a system and method for monitoring disk storage capacity deployed in a containerized manner are used, Prometheus is used to collect and store monitoring data, and a flexible alarm mechanism is implemented through Alertmanager; the host non-mount point disk storage capacity information is collected by developing Exporter, and the middleware service of the host disk is monitored; the disk capacity situation is monitored at any time through a visual monitoring dashboard to prevent abnormal operation of system services caused by insufficient disks; and the efficiency and reliability of middleware management are improved by monitoring the non-mount point disk storage capacity of the containerized middleware Pod using the Hostpath storage mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural diagram of a system for monitoring disk storage capacity in containerized deployment.

[0016] Figure 2 The present invention is a flowchart of a method for containerized deployment of disk storage capacity monitoring. DETAILED DESCRIPTION

[0017] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described with reference to the accompanying drawings.

[0018] As attached Figure 1 As shown, a system for monitoring disk storage capacity in containerized deployment includes a data indicator collection module, a data indicator calculation module, and a data indicator exposure module; The data indicator collection module collects non-mount point disk capacity numerical indicators used by the containerized deployment middleware; The data indicator calculation module calculates the non-mount point disk capacity numerical indicator used by the containerized deployment middleware; The data indicator exposure module exposes the non-mount point disk capacity numerical indicator used by the containerized deployment middleware in an http manner.

[0019] Furthermore, a system for monitoring disk storage capacity in containerized deployment is provided, wherein the function of the data indicator collection module is to determine whether a directory exists based on the disk directory mounted by the container pod, and if the disk directory exists, a system command is called to collect the disk capacity value returned by the system command.

[0020] Further, in a system for monitoring disk storage capacity in containerized deployment, the data indicator calculation module functions to perform unit calculation and conversion based on the data value returned by the data indicator collection module; The units include KB, MB, and GB.

[0021] Furthermore, in a system for monitoring disk storage capacity in a containerized deployment, the function of the data indicator exposure module is to name the values ​​calculated by the data indicator calculation module and encapsulate them as Prometheus indicators for exposure.

[0022] As attached Figure 2 As shown, a method for containerized deployment of disk storage capacity monitoring includes the following sub-steps: S1: Create a Docker image and set up pre-configured non-mount point disk capacity data collection applications and related tools; S2: Use Golang to write an application Exporter that collects non-mount point disk capacity data and generates Prometheus monitoring indicators; S3: Exposes indicators through Http, and the Prometheus client library pushes data to the Prometheus service; S4: Deploy the application for collecting non-mount point disk capacity data to each host that needs to collect non-mount point disk storage capacity data through the K8s Daemonset mode, and check the service operation status; S5: Configure Prometheus to collect non-mount point disk storage capacity indicator data; S6: Configure the Grafana dashboard and write PromQL to display the non-mount point disk storage capacity indicator data; S7: Monitor the non-mounted disk capacity through the containerized middleware non-mounted disk capacity data collection system.

[0023] Furthermore, a method for monitoring disk storage capacity in containerized deployment is provided, wherein the related tools include a configuration file for collecting non-mount point disk capacity data, an initialization script, and necessary dependencies.

[0024] Specific embodiment 1 Docker: Image: The image is the basic component of the Docker container. It contains a complete operating environment, including the operating system, applications, and dependencies. The image is read-only and can be obtained and shared through the Docker image repository.

[0025] Container: A container is a runnable instance created based on an image. The container provides an isolated operating environment that enables applications and their dependencies to run in a consistent manner in different computing environments. The container is orchestrated, portable, and reusable.

[0026] Registry: A registry is a place for storing and sharing Docker images. Docker Hub is a commonly used public registry where users can obtain official and community-shared images. Users can also build private registry to store and manage their own images.

[0027] Dockerfile: Dockerfile is a text file that defines the image building process. By writing Dockerfile, you can specify the base image, install software, configure environment variables, copy files, and other operations to build a custom image.

[0028] Specific embodiment 2 Kubernetes: Container orchestration: Kubernetes allows users to define and manage the deployment, scheduling, and scaling of multiple containerized applications. It can automatically allocate and schedule containers on nodes in the cluster to ensure high availability and load balancing of applications.

[0029] Storage orchestration: Hostpath mode, using host disks.

[0030] Specific embodiment 3 Prometheus: Metrics: The basic unit of Prometheus monitoring is metrics. Each metric consists of a name and a set of key-value pairs, labels or label values. These labels allow fine-grained classification and filtering of metrics by label values ​​without changing the metric name.

[0031] Query language PromQL: Prometheus provides a powerful query language called PromQL for querying and aggregating time series data. Users can use PromQL to generate charts, calculate alert conditions, and more.

[0032] Scraping: Prometheus regularly scrapes indicator data through HTTP endpoints. The HTTP endpoints are called targets. The Prometheus configuration file defines the targets to be scraped and the frequency of scraping.

[0033] Specific Example 4 Grafana: Dashboard: The core of Grafana is the dashboard. A dashboard is a collection of one or more panels that display data from different data sources. Users can create custom dashboards by dragging and dropping components, configuring queries, and setting chart styles.

[0034] Data source: The data source is the backend system that Grafana uses to obtain data. Grafana supports multiple data sources, including Prometheus, InfluxDB, Elasticsearch, MySQL, etc. Users can configure multiple data sources and use them in the dashboard.

[0035] Panel: Panel is the basic visualization component in Grafana, which is used to display specific data. Panel types include line charts, bar charts, heat maps, tables, statistical charts, etc. Each panel can obtain data from one or more data sources and use query language to query data.

[0036] Specific Example 5 The data index collection and calculation code is: func NewDirCollector(options DirCollectorOption) (*NamedDirCollector,error) { cli, err := client.NewClientWithOpts(client.FromEnv) if err != nil { return nil, err } p :=&NamedDirCollector{ scrapeChan: make(chan scrapeRequest), source: cli, dirPath: options.DirPath, debug: options.Debug, } go p.start() return p, nil } ... func (p *NamedDirCollector) scrape(ch chan<- prometheus.Metric) { var sendMetrics = true subdirectories, err := disk.GetImmediateSubdirectories(p.dirPath) if err != nil || subdirectories == nil { sendMetrics = false log.Printf("error reading dirs: %v", err) }} podNameDir , err := containers . GetPlatformPodName ( subdirectories , p . source ) if err != nil || podNameDir == nil { sendMetrics = false log.Printf("error reading containers: %v", err) }} dirCapacity , err := disk . GetDir ( subdirectories ) ; if err != nil || dirCapacity == nil { { sendMetrics = false.

[0037] Other than the 6-year-old snowflake var ( diskCapacityDesc = prometheus .NewDesc( "disk_capacity" "Disk capacity" []string{"path", "hostname", "podname"}, nil) ) ... if !sendMetrics { . log.Printf("error reading unmount dir capacity: %v", error) } else { log.Printf("reading unmount dir capacity") for path := range dirCapacity { for hostPort := range podNameDir { if path == hostPort { ch<- prometheus.MustNewConstMetric(diskCapacityDesc, prometheus.GaugeValue,float64(dirCapacity[path]), path, podNameDir[hostPort].HostName, podNameDir[hostPort].PodName) continue } } }.

[0038] This solution uses a system and method for monitoring disk storage capacity in containerized deployment, uses Prometheus to collect and store monitoring data, and implements a flexible alarm mechanism through Alertmanager; collects host non-mount point disk storage capacity information by developing Exporter, and monitors the middleware service of the host disk; keeps an eye on the disk capacity at any time through a visual monitoring dashboard to prevent system service operation abnormalities caused by insufficient disk; and improves the efficiency and reliability of middleware management by monitoring the non-mount point disk storage capacity of the containerized middleware Pod using the Hostpath storage mode.

[0039] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A system for monitoring disk storage capacity in containerized deployment, characterized in that: It includes data indicator collection module, data indicator calculation module and data indicator exposure module; The data indicator collection module collects non-mount point disk capacity numerical indicators used by the containerized deployment middleware; The data indicator calculation module calculates the non-mount point disk capacity numerical indicator used by the containerized deployment middleware; The data indicator exposure module exposes the non-mount point disk capacity numerical indicator used by the containerized deployment middleware in an http manner.

2. According to the system for monitoring the storage capacity of a containerized disk deployment according to claim 1, it is characterized in that: The function of the data indicator collection module is to determine whether the directory exists based on the disk directory mounted by the container pod. If the disk directory exists, the system command is called to collect the disk capacity value returned by the system command.

3. According to the system for monitoring the storage capacity of a containerized disk deployment as claimed in claim 1, it is characterized in that: The function of the data indicator calculation module is to perform unit calculation and conversion based on the data value returned by the data indicator collection module; The units include KB, MB, and GB.

4. According to the system for monitoring the storage capacity of a containerized disk deployment as claimed in claim 1, it is characterized in that: The function of the data indicator exposure module is to name the values ​​calculated by the data indicator calculation module and encapsulate them as Prometheus indicators for exposure.

5. A method for monitoring disk storage capacity in containerized deployment, implemented based on a system for monitoring disk storage capacity in containerized deployment as claimed in any one of claims 1 to 4, characterized in that: It includes the following sub-steps: S1: Create a Docker image and set up pre-configured non-mount point disk capacity data collection applications and related tools; S2: Use Golang to write an application Exporter that collects non-mount point disk capacity data and generates Prometheus monitoring indicators; S3: Exposes indicators through Http, and the Prometheus client library pushes data to the Prometheus service; S4: Deploy the application for collecting non-mount point disk capacity data to each host that needs to collect non-mount point disk storage capacity data through the K8s Daemonset mode, and check the service operation status; S5: Configure Prometheus to collect non-mount point disk storage capacity indicator data; S6: Configure the Grafana dashboard and write PromQL to display the non-mount point disk storage capacity indicator data; S7: Monitor the non-mounted disk capacity through the containerized middleware non-mounted disk capacity data collection system.

6. A method for monitoring disk storage capacity in containerized deployment according to claim 5, characterized in that: The related tools include configuration files, initialization scripts, and necessary dependencies for collecting non-mount point disk capacity data.

Images