A method and system for managing and monitoring a database connection pool

Abstract

The application relates to a database connection pool management and monitoring method and system, which is used for managing and monitoring database connection pools of multiple clients, and comprises the following steps: S1, collecting data of multiple database connection pools through a data probe JavaAgent; S2, forwarding the data of the database connection pools to a stream processing module after the data is summarized; S3, the stream processing module carries out cleaning and clustering processing on the data of the database connection pools, and stores the processed data into a storage module; S4, an alarm module is used for alarming and early warning the data in the storage module; and S5, a server is used for acquiring a configuration version of the database connection pool in the data of the storage module, and updating the database connection pool configuration to the latest configuration. Compared with the prior art, the application is accurate and efficient, can monitor multiple database connection pools, and can realize dynamic adjustment of the connection pool configuration.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of connection pool monitoring, in particular to a database connection pool management and monitoring method and system. BACKGROUND

[0002] In existing engineering practice, the connection for accessing the database is basically managed by using a connection pool, and common database connection pools on the market include C3P0, DBCP, Druid, and hikari.

[0003] For the monitoring of the running state of the connection pool, the common JMX data collection and monitoring method requires that the database connection pool itself implements the JMX interface, but the implementation methods of each database connection pool are different, the required configurations are different, and even some database connection pools do not implement the JMX interface; therefore, to uniformly monitor and manage the database connection pool, additional configurations of the managed database connection pool are required, and even the code needs to be modified; the more the number and types of managed connection pools are, the higher the complexity of the collection end is.

[0004] For the management of the connection pool parameters, the scheme of restarting the service to make the new parameters take effect has a great impact on the continuity and availability of the service, and the long period of taking effect of the new parameters is also a problem; and the springcloud config hot loading scheme requires that the service is implemented using the springcloud technology, and the scheme reinitializes the entire spring container, which has the problems of slow initialization, initialization of irrelevant parts of the database connection pool, and large scope. SUMMARY

[0005] The present application relates to the field of connection pool monitoring, in particular to a database connection pool management and monitoring method and system.

[0006] The object of the present application can be achieved by the following technical solutions.

[0007] A database connection pool management and monitoring method for managing and monitoring a plurality of client database connection pools, comprising the following steps:

[0008] S1: collecting data of a plurality of database connection pools through a data probe JavaAgent;

[0009] S2: aggregating and forwarding the data of the database connection pools to a stream processing module;

[0010] S3: the stream processing module performs cleaning and clustering processing on the data of the database connection pools, and stores the processed data in a storage module;

[0011] S4: the data in the storage module is alarmed and warned by the alarm module;

[0012] S5: the configuration version of the database connection pool in the data of the storage module is obtained by the server, and the database connection pool configuration is updated to the latest configuration.

[0013] Preferably, the data probe JavaAgent directly accesses the database connection pool to obtain data.

[0014] Preferably, the data obtained by the data probe JavaAgent includes data source information, execution information and sql information.

[0015] Preferably, in step S2, the data collected by the data probe JavaAgent is summarized by the Fluentbit component, and then sent to the Kafka component for caching, and then forwarded to the streaming processing module through the Kafka message queue.

[0016] Preferably, the alarm module includes an alarm component and a warning component, the alarm component includes automatically generating an alarm rule configuration file through an elasalert alarm framework, and sending it to a remote server for running, realizing visual page configuration management alarm rule, and then issuing to a remote server for execution.

[0017] The pre-warning component compares the data before and after the pre-warning field collected, and pre-warns when the data changes exceed the pre-warning range.

[0018] Preferably, when the server updates the database connection pool configuration, the data probe JavaAgent uses the javassist bytecode injection technology to dynamically modify the bytecode file when the JVM loads the class bytecode file, and injects the logic code of data collection and the logic code of configuration hot loading, to complete the update of the configuration.

[0019] A database connection pool management and monitoring system for managing and monitoring a plurality of client database connection pools, the management and monitoring system comprising a client acquisition module, a message forwarding module, a streaming processing module, a storage module, an alarm module, a server,

[0020] The client acquisition module acquires data of a plurality of database connection pools through a data probe JavaAgent,

[0021] The message forwarding module forwards the data of the database connection pool to the streaming processing module after summarizing the data,

[0022] The streaming processing module cleans and clusters the data of the database connection pool, and stores the processed data in the storage module.

[0023] The alarm module alarms and prewarns the data in the storage module,

[0024] The server is used for accessing the data in the storage module, obtaining the configuration version of the database connection pool in the data of the storage module, and updating the connection pool configuration to the latest configuration.

[0025] Preferably, the data probe JavaAgent directly accesses the database connection pool to obtain data.

[0026] Preferably, when the server updates the database connection pool configuration, the data probe JavaAgent uses the javassist bytecode injection technology to dynamically modify the bytecode file when the JVM loads the class bytecode file, injects the logic code of data collection and the logic code of configuration hot loading, and completes the update of the configuration.

[0027] Preferably, the alarm module includes an alarm component and a prewarning component, the alarm component includes automatically generating an alarm rule configuration file through an elasalert alarm framework and sending it to a remote server for running, realizing visual page configuration management of alarm rules, and then issuing it to the remote server for execution.

[0028] The prewarning component compares the data before and after the prewarning field collected, and prewarns when the data changes exceed the prewarning range.

[0029] Compared with the prior art, the present application has the following advantages:

[0030] (1) The present application directly obtains the data of the database connection pool by using the data probe JavaAgent, effectively collects and obtains the data of multiple database connection pools, accurately collects in collection, lightly removes and compresses the collected data, does not have too high consumption on network traffic, and forwards storage, which is convenient for subsequent management and monitoring, manages the running state of each database connection pool, discovers the running exception of the database connection pool in time, and prevents the connection storm of the database.

[0031] (2) The server of the present application can obtain the parameter version of the database connection pool in real time, dynamically adjusts the version of the database connection pool, dynamically validates the new adjustment of the connection pool, does not need to restart the application, validates quickly, does not affect other irrelevant parts, improves the resource utilization rate, and improves the continuity and availability of the service.

[0032] (3) The application places the logical code of data collection and configuration hot loading in the JavaAgent through the data probe JavaAgent, injects the logical code into the code of the application through the bytecode injection technology when the application starts, and does not need to modify the code of the application end and special configuration requirements, so that the dynamic adjustment of data collection and connection pool configuration is realized in the scene of non-invasion to the application. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The flowchart of the application is shown in the figure.

[0034] Figure 2 The data flow diagram of the application is shown in the figure.

[0035] Figure 3 The flowchart of the connection pool configuration update in the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0036] The application will be described in detail below in combination with the drawings and specific embodiments. It should be noted that the following description of the embodiments is only substantially exemplary, and the application is not intended to limit the application or its use, and the application is not limited to the following embodiments.

[0037] EMBODIMENT

[0038] A database connection pool management and monitoring method, as shown in Figure 1 , 2 , is used for managing and monitoring the database connection pool of a plurality of clients, and includes the following steps:

[0039] S1: Collecting the data of a plurality of database connection pools through a data probe JavaAgent.

[0040] The data probe JavaAgent of the application directly accesses the database connection pool to obtain data, and the obtained data includes data source information, execution information and sql information.

[0041] In specific implementation, when the database connection pool in the application needs to be monitored, only the parameter of the JavaAgent needs to be added when the application starts, the jar package is referenced, and the data can be collected after normal start. The data collection of the JavaAgent is realized by directly accessing the database connection pool, obtaining the data source information, execution information and sql information from the database connection pool, and then actively pushing the data to the remote server.

[0042] S2: The data of the database connection pool is summarized and forwarded to a stream processing module.

[0043] The data collected by the data probe JavaAgent is summarized by the Fluentbit component, and then sent to the Kafka component for caching, and then forwarded to the stream processing module through the Kafka message queue.

[0044] S3: The stream processing module cleans and clusters the data of the database connection pool, and stores the processed data in the storage module.

[0045] The stream processing module performs real-time stream data statistical calculation according to the data of the accessed Kafka topic. First, the stream processing module accesses data according to different Topic Input, and as long as there is new data in the kafka topic, it will automatically flow into the stream processing module. Each processing module accesses each entity class that needs to be processed through the Kstream method, and then stores the received entity object through Flatmap The value of <K, V> is convenient for statistical calculation and grouping sorting.

[0046] The stream processing module includes two steps, first, through Database-aggregation-processor, Instance-aggregation-processor, datasource-to-init-transformer, datasource-to-instance transformer, datasource-to-execute transformer, sql-info-transformer, the metadata of instance is counted, including counting the number of instances contained by all projects under the application, the number of instances contained by all databases, and the statistical values of various indicators, etc. Six modules will uniformly call various entity classes defined in the common-modle to receive data. After processing, grouping, and statistical operation, sink sends the downstream module. Including http sink to api-server interface for front-end display, es sink to elasticsearch database for persistent storage of index data, jdbc sink to mysql for persistent storage of index metadata.

[0047] In the present application, the data processed by the stream processing module is forwarded to the storage module through the Fluentbit component. The storage module of the present application is Figure 2 es cluster module.

[0048] S4: The data in the storage module is alarmed and warned through the alarm module.

[0049] The alarm module comprises an alarm component and a pre-warning component, the alarm component comprises automatically generating an alarm rule configuration file through an elasalert alarm framework and sending the alarm rule configuration file to a remote server to run, realizing visual page configuration management of the alarm rule and then issuing the alarm rule to the remote server for execution; the pre-warning component compares the front and rear data of the collected pre-warning field, and when the data change exceeds the pre-warning range, pre-warning is performed, for example, the pre-warning of the current connection number is set as when the current connection number reaches 80% of the maximum connection number, the pre-warning of “the connection number is about to reach the upper limit” is triggered.

[0050] The implementation of the pre-warning depends on the analysis and processing of the data change of the index value, and the Kmean clustering algorithm is adopted for the received data. The Kmean clustering algorithm is a K-means clustering algorithm, the logic of which is to divide the data into K groups, select K random clustering centers, and group the objects closest to each clustering center, after the K groups are determined, the logic of clustering processing is still used in each group until no clustering center changes, and the clustering center at this time is obtained. Then, the linear regression algorithm is used to predict the trend of the data, and once the threshold is exceeded, the pre-warning is triggered.

[0051] S5: Obtain the configuration version of the database connection pool in the data of the storage module through the server, and update the database connection pool configuration to the latest configuration.

[0052] The data probe JavaAgent obtains the configuration version of the database connection pool, and the server manager module in Figure 2 compares the configuration version in the storage module with the latest configuration version, and if the configuration version of the database connection pool in the storage module is lower than the latest configuration version, the configuration version is updated. The present application uses the javassist bytecode injection technology of the data probe JavaAgent to dynamically modify the bytecode file when the JVM loads the class bytecode file, injects the logic code of data collection and the logic code of configuration hot loading, and completes the update of the configuration.

[0053] Taking the Druid connection pool as an example, the configuration hot loading is as follows Figure 3 : The DataSourceClassFileTransformer rewrites the transform method by inheriting the ClassFileTransformer, judges when the className is “com / alibaba / druid / pool / DruidDataSource”, and executes the modified instance method after the runtime construction method, realizes the dynamic modification action through javassist dynamic compilation and dynamic execution.

[0054] Corresponding to the method of the application, the application also provides a database connection pool management and monitoring system for managing and monitoring database connection pools of multiple clients, the management and monitoring system comprising a client acquisition module data probe JavaAgent, a message forwarding module, a streaming processing module, a storage module, an alarm module, and a server, the data probe JavaAgent being used to acquire data of multiple database connection pools, the message forwarding module being used to forward the data of the database connection pools to the streaming processing module after summarizing the data, the streaming processing module being used to clean and cluster the data of the database connection pools and store the processed data in the storage module, the alarm module being used to alarm and warn the data in the storage module, and the server being used to access the data in the storage module, acquire the configuration version of the database connection pool in the data of the storage module, and update the connection pool configuration to the latest configuration.

[0055] Specifically, the data probe JavaAgent directly accesses the database connection pool to acquire data. When the server updates the database connection pool configuration, the data probe JavaAgent uses the javassist bytecode injection technology to dynamically modify the bytecode file when the JVM loads the class bytecode file, injects the logic code of data acquisition and the logic code of configuration hot loading, and completes the update of the configuration. The alarm module comprises an alarm component and a warning component, the alarm component comprises an elasalert alarm framework for automatically generating an alarm rule configuration file and sending it to a remote server for running, realizing visual page configuration management of alarm rules, and then issuing them to the remote server for execution, and the warning component compares the data before and after the warning field collected, and performs warning when the data changes exceed the warning range.

[0056] The above embodiments are only examples and do not limit the scope of the application. These embodiments can also be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the technical idea of the application.

Claims

1. A method of managing and monitoring a database connection pool, characterized by, A management and monitoring system for a plurality of client database connection pools, comprising the following steps: S1: collecting data of a plurality of database connection pools by a data probe JavaAgent; the data probe JavaAgent directly accesses the database connection pools to obtain data; S2: aggregating the data of the database connection pools and forwarding the aggregated data to a stream processing module; S3: the stream processing module performs cleaning and clustering processing on the data of the database connection pools, and stores the processed data in a storage module; S4: the alarm module alarms and warns the data in the storage module; S5: the server obtains the configuration version of the database connection pool in the data in the storage module, and updates the database connection pool configuration to the latest configuration; When the server updates the database connection pool configuration, the data probe JavaAgent uses the javassist bytecode injection technology to dynamically modify the bytecode file when the JVM loads the class bytecode file, and injects the logic code of data collection and the logic code of configuration hot loading to complete the update of the configuration.

2. The method for managing and monitoring a database connection pool according to claim 1, wherein, The data obtained by the data probe JavaAgent includes data source information, execution information and sql information.

3. The method of claim 1, wherein the method further comprises: In step S2, the data collected by the data probe JavaAgent is aggregated by a Fluentbit component, and then sent to a Kafka component for caching, and then forwarded to the stream processing module through the Kafka message queue.

4. The method of claim 1, wherein the database connection pool is managed by a database management system. The alarm module includes an alarm component and a warning component, the alarm component includes generating an alarm rule configuration file automatically through an elasalert alarm framework, and sending the alarm rule configuration file to a remote server for running, realizing visual page configuration management alarm rule, and then issuing to the remote server for execution; The pre-warning component compares the data before and after the pre-warning field is collected, and pre-warns when the data changes exceed the pre-warning range.

5. A management monitoring system for a database connection pool, characterized by A management and monitoring system for a plurality of client database connection pools, the management and monitoring system comprising a client collection module, a message forwarding module, a stream processing module, a storage module, an alarm module, a server, The client collection module collects data of a plurality of database connection pools by a data probe JavaAgent, the data probe JavaAgent directly accesses the database connection pools to obtain data, The message forwarding module aggregates the data of the database connection pools and forwards the aggregated data to the stream processing module, The stream processing module performs cleaning and clustering processing on the data of the database connection pools, and stores the processed data in the storage module, The alarm module alarms and warns the data in the storage module, The server is used to access the data in the storage module, obtain the configuration version of the database connection pool in the data in the storage module, and update the connection pool configuration to the latest configuration, When the server updates the database connection pool configuration, a data probe JavaAgent uses a javassist bytecode injection technology to dynamically modify a bytecode file when a class bytecode file is loaded by the JVM, injects logic code for data collection and logic code for configuration hot loading, and completes updating of the configuration.

6. The system for managing and monitoring of database connection pool according to claim 5, wherein, The alarm module comprises an alarm component and a pre-alarm component, the alarm component comprises an alarm rule configuration file automatically generated through an elasalert alarm framework and sent to a remote server for running, realizing visual page configuration management of alarm rules and then issuing to the remote server for execution; The pre-alarm component compares the front and back data of the collected pre-alarm field, and pre-alarms when the data change exceeds the pre-alarm range.

Images