Distributed database built-in monitoring system

Abstract

The invention discloses a distributed database built-in monitoring system, belongs to the technical field of databases, and aims to solve the technical problem of how to overcome the defects of complex monitoring deployment, high data response delay, coarse information acquisition granularity and low system integration in the prior art. According to the technical scheme, a monitoring platform module is integrated with a database kernel, a front-end and rear-end separation framework is adopted, and visual display and centralized management and control of the database operation state are achieved; the data acquisition module is used for simultaneously starting with a database process through a built-in HTTP (Hyper Text Transport Protocol) server to realize native support of a monitoring function; the distributed state query module is used for discovering cluster node states through a Gossip protocol, supporting node health detection and remote calling state service, and supporting gRPC-based streaming communication and concurrent processing; and the time sequence data management module is used for storing, compressing and querying the collected time sequence monitoring data.

Description

Technical Field

[0001] This invention relates to the field of database technology, specifically to a distributed database built-in monitoring system. Background Technology

[0002] With the development of database technology and distributed systems, more and more business systems are adopting distributed databases to support the storage and querying needs of large-scale data. Compared with traditional centralized databases, distributed databases have significant advantages in scalability, fault tolerance, and multi-node concurrent processing capabilities. However, as system complexity increases, the operational status of distributed databases becomes more difficult to monitor and manage.

[0003] Current database monitoring solutions primarily rely on externally deployed monitoring platforms, such as Prometheus+Grafana and Zabbix. While these solutions offer some general applicability, they suffer from the following issues:

[0004] ① High deployment complexity: External monitoring platforms usually require separate installation, configuration and maintenance, lack tight integration with database systems, and have high operation and maintenance costs;

[0005] ② Large data latency and coarse collection granularity: Due to the reliance on external collectors to periodically pull metrics, there is a collection latency, and it is difficult to obtain fine-grained metadata information from within the database;

[0006] ③ Lack of deep semantic information: External platforms cannot directly access the database's internal system tables and executors, making it difficult to provide rich management information such as permissions, table structure, and statement execution.

[0007] ④ Inconsistent interfaces and insufficient scalability: Traditional monitoring solutions have scattered data interfaces, which are not conducive to system-level expansion or integration with other management systems.

[0008] Furthermore, in distributed environments, complex inter-node communication and frequent state changes place higher demands on the system's real-time performance and consistency. Traditional polling-based data collection methods are insufficient to meet the practical needs of large-scale distributed databases for low latency, high throughput, and real-time diagnostics.

[0009] Therefore, how to overcome the shortcomings of existing technologies, such as complex monitoring deployment, high data response latency, coarse information acquisition granularity, and low system integration, and achieve native, real-time, efficient, and scalable monitoring and management of the database system's operating status is a technical problem that urgently needs to be solved. Summary of the Invention

[0010] The technical objective of this invention is to provide a distributed database built-in monitoring system to overcome the shortcomings of existing technologies, such as complex monitoring deployment, high data response latency, coarse information acquisition granularity, and low system integration, and to achieve native, real-time, efficient, and scalable monitoring and management of the database system's operating status.

[0011] The technical objective of this invention is achieved as follows: a distributed database built-in monitoring system, which includes a monitoring platform module, a data acquisition module, a distributed status query module, a time-series data management module, and an interface conversion module.

[0012] The monitoring platform module is integrated with the database kernel and adopts a front-end and back-end separation architecture to realize the visualization and centralized management of the database's operating status;

[0013] The data acquisition module is used to start simultaneously with the built-in HTTP server and database process to achieve native support for monitoring functions. It uses an internal executor to execute SQL statements and access system interfaces, and collects database permissions, table structure, engine type and region configuration information, and caches the collection results in memory.

[0014] The distributed status query module is used to discover the status of cluster nodes through the Gossip protocol, supports node health detection and remote invocation of status services, and supports streaming communication and concurrent processing based on gRPC to improve status query efficiency.

[0015] The time-series data management module is built on the underlying KV engine and is used to store, compress, and query the collected time-series monitoring data.

[0016] The interface conversion module is used to automatically convert gRPC interfaces and RESTful APIs based on gRPC-gateway, enabling clients to access monitoring data via HTTP protocol and supporting cross-language calls and interface security control mechanisms. The interface security control mechanism includes authentication and authorization control of the visitor's identity, requiring administrator privileges for all management operations; for returned information containing sensitive fields, data anonymization is automatically performed to ensure information security.

[0017] As a preferred embodiment, the monitoring platform module includes a front-end monitoring interface sub-module and a back-end service sub-module;

[0018] The front-end monitoring interface submodule is built on a web framework. The compiled static resource files are integrated into the database service program through an embedding mechanism, that is, embedded into the database executable file in the form of static resources, to achieve seamless system-level integration. The front-end monitoring interface submodule accesses the database embedded static resources through a browser and interacts with the back-end service through a RESTful interface to obtain real-time monitoring data.

[0019] The backend service submodule is developed and built based on a high-performance programming language and the gRPC protocol. It is internally divided into several core services, including database metadata management service, cluster status query service, and time-series data management service. The services communicate with each other through a unified interface protocol. Service registration and initialization are completed synchronously during the database startup phase, thereby ensuring the availability and real-time performance of monitoring.

[0020] As a preferred approach, the data acquisition module executes database instructions through its internal executor to obtain core metadata information covering database permissions, table structure, engine type, regional distribution, columns, and indexes; at the same time, it accesses the underlying descriptor information by calling the database's internal interface to obtain descriptor ID information, permission configuration information, column and index information, thus avoiding the additional performance overhead caused by traditional SQL statement parsing.

[0021] For detailed information on table structure and indexes, a combination of parsing storage paths and executing built-in query commands is used to obtain the information, and in special cases, extended information including memory mapping and partitions is collected; permission configuration information is obtained by periodic scanning combined with event listening to ensure real-time response to permission changes.

[0022] All data collected by the data acquisition module is cached in memory, enabling efficient data access and management.

[0023] As a preferred approach, the distributed status query module discovers that each node reports its own status information via broadcast, and by parsing the node identification information and its liveness status, it achieves real-time perception of the overall health status of the cluster.

[0024] During the status query process, the distributed status query module automatically determines the attributes of the target node being requested.

[0025] If it is a local node, it directly accesses the system module to obtain status information;

[0026] If it is a remote node, it requests status services through remote communication. Remote requests use an efficient streaming communication protocol to support the batch return of large amounts of status information, thereby improving transmission efficiency.

[0027] As a preferred embodiment, when performing status queries on remote nodes, the distributed status query module automatically determines the node's affiliation based on the nodeID and establishes a connection using the GRPCDialNode method.

[0028] As a preferred approach, the system adopts a parallel processing architecture. During status queries, it ensures query performance in a large-scale node environment through multi-coroutine scheduling and a unified result collection mechanism.

[0029] As a preferred embodiment, the data acquisition module periodically reads raw monitoring data from the system status and stores the sampled data in the KV storage system at predetermined time intervals. The data storage structure adopts a unified key-value naming method to facilitate indexing and retrieval. The KV storage system adopts a hot and cold tiered strategy, with hot data cached in memory and cold data archived in the underlying storage. Data expiration is cleaned up through a background thread to ensure data validity and performance stability during long-term system operation.

[0030] As a preferred option, the system periodically compresses historical time-series data, uses an aggregation algorithm to generate high-time-granularity summary data, improves storage efficiency, and supports data acquisition with a Resolution 10s precision. The compression algorithm supports multiple aggregation methods, including average, maximum, and minimum values.

[0031] As a preferred option, to enhance the system's compatibility and openness, the system uses a protocol gateway component to convert high-performance gRPC services into general RESTful API interfaces. The RESTful API interface definitions are managed through a unified description file, supporting multi-language calls and direct access from the front end. The gateway layer supports multiplexing, automatic data format conversion, and general header information injection, while also having request timeout control and permission verification mechanisms.

[0032] Even better, the interface conversion module adds generic request header information to each request during the conversion process and supports JSON and protobuf format responses.

[0033] The distributed database built-in monitoring system of the present invention has the following advantages:

[0034] (i) This invention is deeply integrated with the main database process, without relying on additional monitoring components. It automatically initializes and runs during the database startup process, which greatly simplifies the operation and maintenance deployment process and achieves the goal of built-in integration and simple deployment.

[0035] (ii) By directly calling the internal executor and system interface, the system can collect database metadata, permissions, table structure, statement execution and other information in real time. The monitoring granularity is fine and the response latency is low, which is suitable for highly dynamic database environments.

[0036] (III) This invention achieves node status discovery through the Gossip protocol and combines streaming communication and concurrent processing mechanisms to realize efficient status query and processing of multiple nodes in a large-scale database cluster, thereby improving the overall system monitoring throughput and high-concurrency distributed query capabilities.

[0037] (iv) The present invention has a built-in time series data storage module that supports high-precision data acquisition and compression. Combined with multi-level caching and expiration cleanup strategies, it effectively reduces storage overhead and improves query performance. It does not require the introduction of external time series database components and realizes lightweight time series data management.

[0038] (v) This invention automatically converts gRPC interfaces into RESTful APIs through gRPC-gateway, which not only retains the high performance characteristics of gRPC communication, but also facilitates access by front-end systems, automation tools or other language clients, achieving good scalability and compatibility, and has the ability to provide a unified interface and strong cross-language compatibility.

[0039] (vi) All interfaces of this invention have strict access control and permission verification mechanisms. Sensitive fields are returned with desensitization by default, which effectively protects data security. It is suitable for database management scenarios with high security requirements, realizes security and controllability and supports permission isolation and desensitization, and thus realizes database runtime monitoring capabilities that are deployment-friendly, high-performance and secure. It has good applicability and promotion value in actual production environments.

[0040] (vii) This invention is embedded in the database kernel and is deeply coupled with the database operation process. From the aspects of architecture design, data acquisition, distributed communication, time-series data management and interface provision, it builds a set of stable and timely monitoring solutions to achieve comprehensive monitoring and efficient management of the database operation status.

[0041] (viii) This invention provides a deeply integrated monitoring system embedded within the database system, capable of real-time response. This enhances the maintainability and transparency of the database, lowers the operational threshold for users, and provides reliable, complete, and real-time data support for database operation. It solves problems in existing technologies such as complex monitoring deployment, high data response latency, coarse information acquisition granularity, and low system integration, thereby achieving native, real-time, efficient, and scalable monitoring and management capabilities for the database system's operational status. Specifically, it aims to achieve the following objectives:

[0042] ① Achieve deep integration between the monitoring system and the database, and start running with the database process through an embedded mechanism, without the need to deploy additional external services, thus simplifying operation and maintenance;

[0043] ② Provides comprehensive and fine-grained data collection capabilities, supporting the collection of database permission information, table structure, system indicators, executed statements, metadata information, etc., to improve system observability;

[0044] ③ Construct a high-performance distributed status query mechanism to support real-time query and analysis of the health status and performance data of each node in the cluster;

[0045] ④ Design lightweight time-series data management capabilities to support the collection, compression, cleaning, and multi-granularity aggregation of monitoring data, thereby improving data storage and query efficiency;

[0046] ⑤ Unified interface access method: Improve the efficiency and compatibility of interaction with external systems or front-end interfaces through a dual protocol stack mechanism of gRPC and RESTful interfaces.

[0047] ⑥ Enhance the security and stability of the monitoring system by using mechanisms such as permission verification, sensitive information desensitization, memory control, and abnormal retry to ensure the reliability of system operation.

[0048] ⑦ Through systematic design, a highly integrated, timely, and fully functional database monitoring solution has been achieved, which is suitable for distributed database environments of various sizes;

[0049] (ix) This invention runs synchronously when the database starts through a built-in HTTP service, collects metadata information such as database permissions, table structure, and descriptors, and achieves efficient querying in combination with an internal executor. The distributed state service realizes node discovery and health detection based on the Gossip protocol. It adopts streaming communication and parallel processing technology to improve query efficiency. The underlying lightweight time-series database supports multi-granularity data compression and hierarchical storage. The interface layer converts high-performance gRPC services into RESTful APIs through gRPC-gateway, taking into account cross-language access and system security control.

[0050] (x) The system of the present invention has the advantages of convenient deployment, real-time response and comprehensive functions, which can significantly improve the visibility and operation and maintenance efficiency of distributed databases and is suitable for monitoring and management scenarios of large-scale database systems. Attached Figure Description

[0051] The invention will be further described below with reference to the accompanying drawings.

[0052] Appendix Figure 1 A schematic diagram of the structure of a built-in monitoring system for a distributed database;

[0053] Appendix Figure 2 This is a flowchart illustrating the conversion and access process for a RESTful API interface gateway. Detailed Implementation

[0054] The distributed database built-in monitoring system of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example:

[0056] As attached Figure 1As shown, this embodiment provides a distributed database built-in monitoring system, which includes a monitoring platform module, a data acquisition module, a distributed status query module, a time-series data management module, and an interface conversion module.

[0057] The monitoring platform module is integrated with the database kernel and adopts a front-end and back-end separation architecture to realize the visualization and centralized management of the database's operating status;

[0058] The data acquisition module is used to start simultaneously with the built-in HTTP server and database process to achieve native support for monitoring functions. It uses an internal executor to execute SQL statements and access system interfaces, and collects database permissions, table structure, engine type and region configuration information, and caches the collection results in memory.

[0059] The distributed status query module is used to discover the status of cluster nodes through the Gossip protocol, supports node health detection and remote invocation of status services, and supports streaming communication and concurrent processing based on gRPC to improve status query efficiency.

[0060] The time-series data management module is built on the underlying KV engine and is used to store, compress, and query the collected time-series monitoring data.

[0061] The interface conversion module is used to automatically convert gRPC interfaces and RESTful APIs based on gRPC-gateway, enabling clients to access monitoring data via HTTP protocol and supporting cross-language calls and interface security control mechanisms. The interface security control mechanism includes authentication and authorization control of the visitor's identity, requiring administrator privileges for all management operations; for returned information containing sensitive fields, data anonymization is automatically performed to ensure information security.

[0062] The monitoring platform module in this embodiment includes a front-end monitoring interface sub-module and a back-end service sub-module;

[0063] The front-end monitoring interface submodule is built on a web framework. The compiled static resource files are integrated into the database service program through an embedding mechanism, that is, embedded into the database executable file in the form of static resources, to achieve seamless system-level integration. The front-end monitoring interface submodule accesses the database embedded static resources through a browser and interacts with the back-end service through a RESTful interface to obtain real-time monitoring data.

[0064] The backend service submodule is developed and built based on a high-performance programming language and the gRPC protocol. It is internally divided into several core services, including database metadata management service, cluster status query service, and time-series data management service. The services communicate with each other through a unified interface protocol. Service registration and initialization are completed synchronously during the database startup phase, thereby ensuring the availability and real-time performance of monitoring.

[0065] In this embodiment, the data acquisition module executes database instructions through an internal executor to obtain core metadata information covering database permissions, table structure, engine type, regional distribution, columns, and indexes; at the same time, it accesses the underlying descriptor information by calling the internal database interface to obtain descriptor ID information, permission configuration information, column and index information, thus avoiding the additional performance overhead caused by traditional SQL statement parsing.

[0066] For detailed information on table structure and indexes, a combination of parsing storage paths and executing built-in query commands is used to obtain the information, and in special cases, extended information including memory mapping and partitions is collected; permission configuration information is obtained by periodic scanning combined with event listening to ensure real-time response to permission changes.

[0067] All data collected by the data acquisition module is cached in memory, enabling efficient data access and management.

[0068] In this embodiment, the distributed status query module discovers that each node reports its own status information via broadcast, and by parsing the node identification information and the survival status, it realizes real-time perception of the overall health status of the cluster.

[0069] During the status query process, the distributed status query module automatically determines the attributes of the target node being requested.

[0070] If it is a local node, it directly accesses the system module to obtain status information;

[0071] If it is a remote node, it requests status services through remote communication. Remote requests use an efficient streaming communication protocol to support the batch return of large amounts of status information, thereby improving transmission efficiency.

[0072] In this embodiment, the distributed status query module automatically determines the node's affiliation based on the nodeID when performing status queries on remote nodes, and establishes a connection through the GRPCDialNode method.

[0073] In this embodiment, the system adopts a parallel processing architecture. During status queries, it ensures query performance in a large-scale node environment through multi-coroutine scheduling and a unified result collection mechanism.

[0074] In this embodiment, the data acquisition module periodically reads raw monitoring data from the system status and stores the sampled data in the KV storage system at predetermined time intervals. The data storage structure adopts a unified key-value naming method to facilitate indexing and retrieval. The KV storage system adopts a hot and cold tiered strategy, with hot data cached in memory and cold data archived in the underlying storage. Data expiration is cleaned up through a background thread to ensure data validity and performance stability during long-term system operation.

[0075] In this embodiment, the system periodically compresses historical time-series data, uses an aggregation algorithm to generate high-time-granularity summary data, improves storage efficiency, and supports data acquisition with a Resolution 10s precision. The compression algorithm supports multiple aggregation methods, including average, maximum, and minimum values.

[0076] As attached Figure 2 As shown in this embodiment, in order to improve the system's compatibility and openness, the system converts the high-performance gRPC service into a general RESTful API interface through a protocol gateway component. The RESTful API interface definition is managed through a unified description file, supporting multi-language calls and direct access from the front end. The gateway layer supports multiplexing, automatic data format conversion, and general header information injection functions, and also has request timeout control and permission verification mechanisms.

[0077] In this embodiment, the interface conversion module adds general request header information to each request during the conversion process and supports JSON and protobuf format responses.

[0078] To meet the high performance and high availability requirements of a production environment, this embodiment incorporates several optimizations. For memory management, a tiered monitoring strategy is employed, imposing resource limits on critical path operations to prevent system crashes caused by resource abuse. During query execution, a pre-built execution plan caching mechanism is implemented to reuse execution paths for common query types, reducing compilation overhead.

[0079] Furthermore, this embodiment also incorporates an intelligent retry mechanism, automatically performing a limited number of retries in case of network fluctuations or momentary node unavailability, and controlling the retry frequency using an exponential backoff strategy. To ensure data consistency, critical query and status update operations are processed using a transaction mechanism, ensuring the atomicity of operations in a multi-node concurrent environment.

[0080] The interface conversion module in this embodiment has an administrator permission verification mechanism and automatically desensitizes sensitive fields.

[0081] This embodiment also includes a memory monitoring mechanism to monitor the memory usage of each service module and enable pagination for large-scale query tasks.

[0082] In this embodiment, the distributed state query module uses a coroutine pool and sync.WaitGroup to implement concurrency control, and the query results are collected uniformly through a channel to ensure thread safety.

[0083] This embodiment supports an error retry mechanism, automatically retrying temporary errors and using an exponential backoff strategy to control the retry interval, and supports transaction protection for critical operations.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed database built-in monitoring system, characterized by, The system comprises a monitoring platform module, a data acquisition module, a distributed state query module, a time series data management module and an interface conversion module; The monitoring platform module is integrated with a database kernel, adopts a front-end and back-end separation architecture, and realizes visual display and centralized control of the running state of the database; The data acquisition module is used to start the database process through the built-in HTTP server, realize native support of the monitoring function, execute SQL statements and access system interfaces by using the internal executor, acquire database permissions, table structure, engine type and regional configuration information, and cache the acquisition results in the memory; The distributed state query module is used to discover the cluster node state through the Gossip protocol, support node health detection and remote call state service, and support gRPC-based streaming communication and concurrent processing; The time series data management module is built on the basis of the underlying KV engine, and is used for storing, compressing and querying the acquired time series monitoring data; The interface conversion module is used to realize automatic conversion of gRPC interfaces and RESTful APIs based on grpc-gateway, so that the client accesses the monitoring data through the HTTP protocol, and supports cross-language calling and interface security control mechanism; wherein the interface security control mechanism comprises authentication and authorization control of the visitor identity, and all management operations need to have administrator permissions.

2. The distributed database inbuilt monitoring system of claim 1, wherein, The monitoring platform module comprises a front-end monitoring interface submodule and a back-end service submodule; The front-end monitoring interface submodule is built based on a Web framework, and the generated static resource files are integrated into the database service program through an embedding mechanism, i.e. embedded into the database executable file in the form of static resources, realizing seamless integration at the system level; the front-end monitoring interface submodule accesses the database embedded static resources through a browser, and interacts with the back-end service through a RESTful interface to obtain real-time monitoring data; The back-end service submodule is developed and built based on a high-performance programming language and a gRPC protocol, and is divided into a database metadata management service, a cluster state query service and a time series data management service multiple core services; the services communicate, register and initialize through a uniformly defined interface protocol, and the initialization is completed synchronously in the database startup phase.

3. The distributed database inbuilt monitoring system of claim 1, wherein, The data acquisition module executes database instructions through the internal executor to obtain core metadata information covering database permissions, table structure, engine type, regional distribution, columns and indexes; At the same time, the internal interface of the database is called to access the underlying descriptor information, and descriptor ID information, permission configuration information, column and index information are obtained; For the detailed information of the table structure and the index, a combination of parsing the storage path and executing the built-in query command is adopted to obtain, and in special cases, extended information including memory mapping and partitioning is collected; the permission configuration information is obtained in a periodic scanning combined with event listening manner, to ensure real-time response of permission changes; All the data collected by the data acquisition module is cached in the memory, realizing efficient data access and management.

4. The distributed database inbuilt monitoring system of claim 1, wherein, Each node discovered by the distributed state query module reports its own state information through broadcasting, and realizes real-time perception of the overall health condition of the cluster by analyzing the node identification information and the survival state. The distributed state query module automatically judges the attributes of the target node during the state query process: If it is a local node, directly access the system module to obtain state information; If it is a remote node, request state service through remote communication. Remote requests support batch return of large amounts of state information using efficient streaming communication protocols.

5. The distributed database inbuilt monitoring system of claim 1, wherein, When querying the state of remote nodes, the distributed state query module automatically determines the node's affiliation based on nodeID and establishes a connection through the GRPCDialNode method.

6. The distributed database inbuilt monitoring system of claim 1, wherein, The system uses a parallel processing architecture. During state query, it uses multi-coroutine scheduling and unified result collection mechanism to ensure query performance in large-scale node environment.

7. The distributed database inbuilt monitoring system of claim 1, wherein, The data acquisition module reads raw monitoring data from the system state at regular intervals and stores the sampled data in the KV storage system at predetermined time intervals. The data storage structure uses a unified key-value naming method for easy indexing and retrieval. The KV storage system uses a hot and cold layering strategy, with hot data cached in memory and cold data archived in underlying storage. Background threads are used for data expiration cleaning to ensure data effectiveness and performance stability during long-term system operation.

8. The distributed database inbuilt monitoring system of claim 1, wherein, The system periodically compresses historical time series data, uses aggregation algorithms to generate high-time-granularity summary data, and supports Resolution10s precision data collection. The compression algorithm supports multiple aggregation methods such as average, maximum, and minimum.

9. The distributed database inbuilt monitoring system of claim 1, wherein, The system converts high-performance gRPC services into general RESTful API interfaces through the protocol gateway component. RESTful API interface definitions are managed through unified description files, supporting multi-language calls and direct access from the front end. The gateway layer supports multiplexing, automatic data format conversion, and general header information injection functions, while also having request timeout control and permission verification mechanisms.

10. The in-database monitoring system of any one of claims 1-9, wherein, The interface conversion module adds general request header information to each request during conversion and supports JSON and protobuf format responses.

Images