A Redis-based InfluxDB cluster system and method

Abstract

The present invention discloses a clustering method of InfluxDB based on Redis. When the collected equipment index data is stored in InfluxDB, the data is fragmented and stored in different InfluxDB instances, so as to realize the clustering of InfluxDB, thereby improving the data collection amount and data query performance.

Description

Technical Field

[0001] The present invention relates to an InfluxDB cluster system and method, and specifically to an InfluxDB cluster method based on Redis. The method is mainly used in the field of industrial Internet technology. Background Art

[0002] In the field of industrial Internet, it is necessary to store indicator data collected by industrial equipment. These data are time-series data with characteristics such as huge data volume (hundreds of indicators for a single device) and high collection frequency (at the millisecond level).

[0003] Based on this data characteristic, time series databases are often used in the current field to store such data. Currently, the industry's more well-known time series databases include InfluxDB, OpenTSDB, TDengine, etc., and InfluxDB has been at the top of the time series database list since 2016. At the same time, InfluxDB has the advantages of easy installation, no dependencies, high write performance (supporting 400,000+ data writes per second on a medium-configuration machine); easy calling (native support for HTTP API (Hypertext Transfer Protocol Interface)), easy docking and expansion, and provides SDKs (Software Development Kits) in various languages, SQL-like (Structured Query Language) query languages, time-centric functions, lower learning costs, easier to learn, master and implement; fast query response (within 100ms); support for linked queries and different storage strategies, and has the characteristics and capabilities of fast sampling and hierarchical storage. However, InfluxDB does not natively support distributed cluster capabilities by default, which makes InfluxDB unable to support the high-performance reading and writing requirements of massive indicators. Summary of the invention

[0004] Purpose of the invention: In view of the problems and shortcomings in the prior art, the present invention provides an InfluxDB cluster system and method based on Redis, which can realize writing data to different InfluxDB instances, and accurately obtain data when querying, thereby solving the problem of excessive intrusion into business codes. The business code only needs to call a general method; Redis is used to enable distributed deployed application services to dynamically perceive the current InfluxDB database step-by-step information, and at the same time solve the problem of horizontal expansion of the cluster without shutting down when the amount of data collected by the system increases sharply.

[0005] Technical solution: A cluster system of InfluxDB based on Redis, characterized by comprising a node encapsulation module and a proxy encapsulation module;

[0006] The node encapsulation module: encapsulates the InfluxDB database into a node, which contains the connection information of the InfluxDB database and the metadata table information of the database; encapsulates the operation of the data into a program interface according to the SDK provided by InfluxDB, and the parameter data format of the program interface is consistent with the requirements of the SDK; the node encapsulation module initializes the operating environment when the application starts, including establishing a network connection with InfluxDB and reading the metadata table to obtain the information of the current database storage table, and writing the information of the current database storage table into Redis;

[0007] The proxy encapsulation module: encapsulates the operation of business data into a program interface according to the actual business situation. The data format of the program interface parameters is the format defined in the business system, and the program interface handles the conversion between the business system data format and the SDK data format; the proxy encapsulation module stores the current InfluxDB cluster node information, that is, multiple node information; the proxy encapsulation module contains a blocking queue for data entry to realize the temporary storage of the data entered, and a thread is used to consume the blocking queue data for entry; if the business system uses a distributed configuration center (an independent system that uniformly manages the configuration information of each application), the proxy encapsulation module can dynamically sense the configuration of the InfluxDB cluster and dynamically add the InfluxDB database.

[0008] A clustering method of InfluxDB based on Redis, when storing the collected device indicator data in InfluxDB, the data is sharded and stored in different InfluxDB instances, so as to realize the clustering of InfluxDB, thereby improving the data collection volume and data query performance. The method includes:

[0009] (1) According to the characteristics of equipment indicator data in the industrial Internet field, the indicators of a device model are stored in a data table.

[0010] (2) Each InfluxDB database has a table that records the names of the tables stored in the database, which is referred to as the "metadata table" below. The name of the metadata table is measurement_map.

[0011] (3) When the application starts, it will load the metadata table information in each database according to the configured cluster information, and can obtain the storage status of the data table. At the same time, the storage status of the currently read data table is stored in Redis. The program configures the cluster information of influxDB in the following format: IP: port / database name, IP: port / database name, for example: 172.16.3.8:8086:uiot, 172.16.3.8:8086:uiot2, 172.16.3.8:8086:uiot3. When the application starts, it obtains the configuration information, and can know how many InfluxDB instances are configured, as well as the network connection information of each instance. Then, it connects to the network. After the connection, it reads the metadata table information in each instance through the SDK, and can obtain the data table stored in each instance.

[0012] (4) When the device collects indicator data and puts it into the database, first query whether there is data table information corresponding to the device model in Redis. If it exists, it is directly written to the target table.

[0013] (5) If the data table does not exist, select an InfluxDB based on the load balancing algorithm, then update the metadata table of the target database, write the data table information, and indicate that the data table is stored in the target database; at the same time, write the metadata to the metadata table of the target database. At the same time, update the cache information in Redis. The metadata is used to identify which database the data of the device model exists in.

[0014] (6) When querying device indicator data, first obtain the database instance corresponding to the device model data storage in the cache. If not, return empty directly, otherwise query.

[0015] (7) In addition, for data storage, the producer-consumer model is adopted by default, and the collected indicator data is stored in a blocking queue. The storage task obtains batch data from the blocking queue and distributes it to the respective database nodes according to their respective device models, and then stores it in batches.

[0016] (8) The above operations (3) to (7) are encapsulated into a Proxy module. The Proxy module does not need to be deployed separately. It can be introduced into the InfluxDB application that needs to be operated. When the business code operates the data, it can directly call the program interface provided by the Proxy module.

[0017] The data table name corresponding to the indicator of the device model is "d+device model id".

[0018] The Proxy module is a proxy module for operating InfluxDB.

[0019] Data storage:

[0020] In step S1, the collection task calls the Proxy module to directly store the data into the blocking queue, and returns immediately after storing the data into the queue; the Proxy module refers to the proxy encapsulation module;

[0021] In step S2, the warehousing task of the Proxy module actually warehousing data. It first accesses the metadata table to check whether the data table corresponding to the data exists. If it exists, it directly obtains the corresponding node for operation. Otherwise, it uses the load algorithm to find a database and stores the information of the data table in the metadata table of the database and Redis. In this way, other business application instances can perceive the storage status of the data table.

[0022] Data query:

[0023] When querying data, because the indicators of the device model are stored in a data table, you can know which data table should be queried based on the queried device, and then determine whether the data table already exists. If not, return empty directly, otherwise get the Node where the data table is located for query operations.

[0024] Summary: The InfluxDB operation proxy (Proxy) implemented in the present invention can store data in shards, which can ensure the efficiency of data writing when the amount of data is very large. At the same time, the producer-consumer model adopted further improves the efficiency of data storage; it also supports imperceptible horizontal expansion of the InfluxDB cluster.

[0025] A clustering method of InfluxDB based on Redis, characterized by comprising:

[0026] Data storage:

[0027] In step S1, the collection task calls Proxy to directly store the data into the blocking queue and returns immediately after storing the data into the queue;

[0028] In step S2, the Proxy's warehousing task actually warehousing data. It first accesses the metadata table to check whether the data table corresponding to the data exists. If it exists, it directly obtains the Node of the corresponding node for operation. Otherwise, it uses the load algorithm to find a database and stores the information about the data table in the metadata table of the database ("measurement_map") and Redis. This allows other business application instances to perceive the storage status of the data table.

[0029] Data query:

[0030] When querying data, because the indicators of the device model are stored in a data table, according to the queried device, it is possible to know which data table should be queried, and then determine whether the data table already exists. If not, it directly returns empty, otherwise the Node where the data table is located is obtained for query operations. Beneficial effects: Compared with the prior art, the clustering method of InfluxDB based on Redis provided by the present invention, on the one hand, utilizes the characteristics that the indicator data of the same type of equipment in the field of industrial Internet are mostly the same, divides the data table into libraries, and stores the library information in each database and Redis, so that multiple applications can perceive the data distribution; thereby achieving the purpose of InfluxDB distributed cluster deployment and use; the producer consumer mode and batch warehousing mode used in warehousing can maximize performance. In common device models (300 indicators for one device type), a single node instance can reach 400,000 indicators per second, and combined with the cluster mode, it can easily achieve access to millions of indicator data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is an architecture diagram of a method implementation of an embodiment of the present invention;

[0032] Figure 2 is a flowchart of application startup according to an embodiment of the present invention;

[0033] Figure 3 is a flow chart of data writing in an embodiment of the present invention;

[0034] Figure 4 This is a flow chart of the data actually written into the database according to an embodiment of the present invention;

[0035] Figure 5 This is a flow chart of data query in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0037] A cluster system of InfluxDB based on Redis, including a node encapsulation module and an agent encapsulation module;

[0038] Node encapsulation module: encapsulates the InfluxDB database into a node (Node), which contains the connection information of the InfluxDB database and the metadata table information of the database; encapsulates the data operations into a program interface according to the SDK provided by InfluxDB, and the parameter data format of the program interface is consistent with the SDK requirements; the node encapsulation module initializes the operating environment when the application starts, including establishing a network connection with InfluxDB and reading the metadata table to obtain the information of the current database storage table, and writing the information of the current database storage table into Redis.

[0039] Proxy encapsulation module: According to the actual business situation, the business data operation is encapsulated into a program interface. The data format of the program interface parameters is the format defined in the business system. The program interface handles the conversion between the business system data format and the SDK data format. The proxy encapsulation module stores the current InfluxDB cluster node information, that is, multiple node information. The proxy encapsulation module contains a blocking queue for data entry to realize the temporary storage of data entry, and a thread is used to consume the blocking queue data for entry. If the business system uses a distributed configuration center (an independent system that uniformly manages the configuration information of each application), the proxy encapsulation module can dynamically sense the configuration of the InfluxDB cluster and dynamically add the InfluxDB database.

[0040] The node encapsulation module encapsulates the basic information of the InfluxDB SDK client and database instance; the proxy encapsulation module internally stores all node module encapsulation instances of the InfluxDB cluster, encapsulates the management operations of the InfluxDB cluster nodes and the database routing sharding function for data processing. When in use, node encapsulation is performed first, and then proxy encapsulation.

[0041] Proxy encapsulation: According to the actual business situation, the operations such as adding, deleting, modifying and checking business data are encapsulated, and a program interface is provided. The proxy encapsulation module stores the current InfluxDB cluster node information, that is, multiple node module encapsulation instances; there is a blocking queue for data entry inside the module to realize the temporary storage of the data entry, and there is also a thread (that is, Figure 4 The storage task in the queue) consumes the queue data for storage (this is the producer consumer design pattern); if the business system uses a distributed configuration center (a system used to uniformly manage all application configuration information in the microservice architecture system), it can dynamically perceive the configuration of the InfluxDB cluster and dynamically add the InfluxDB database.

[0042] The connection information refers to the IP address and port where the InfluxDB database is deployed.

[0043] The content in the metadata table is used to record which data tables are stored in the current database.

[0044] According to the characteristics of equipment indicator data in the industrial Internet field, the indicators of a device model are stored in a data table.

[0045] Each InfluxDB database has a metadata table.

[0046] The node encapsulation module encapsulates batch insertion and deletion operations of data into a program interface based on the SDK provided by InfluxDB.

[0047] The proxy encapsulation module encapsulates operations such as adding, deleting, modifying and querying business data into a program interface.

[0048] When the application starts, it will load the metadata table information in each database according to the configured cluster information, obtain the storage status of the data table, and store the currently read data table storage status in Redis.

[0049] A clustering method of InfluxDB based on Redis, the method comprising:

[0050] According to the characteristics of equipment indicator data in the industrial Internet field, the indicators of a device model are stored in a data table with the table name "d+device model id".

[0051] First, each InfluxDB database has a table named "measurement_map", which is referred to as the "metadata table" below. The metadata table records the names of the data tables stored in this database.

[0052] When the application starts, it will load the metadata table information in each database according to the configured cluster information, obtain the storage status of the data table, and store the currently read data table storage status in Redis.

[0053] When the device collects indicator data and puts it into the database, first query whether there is data table information corresponding to the device model in Redis. If it exists, it is directly written to the target table.

[0054] If the data table does not exist, an InfluxDB is selected based on the load balancing algorithm, and metadata (which indicates which database the data of the device model is stored in) is written to the metadata table of the target database. At the same time, the cache information in Redis is updated.

[0055] When querying device indicator data, first obtain the database instance corresponding to the device model data storage in the cache. If not, return empty directly, otherwise query.

[0056] In addition, for data warehousing, the producer-consumer mode is adopted by default, and the collected indicator data is stored in a blocking queue. The warehousing task obtains batch data from the blocking queue and distributes it to the respective database nodes according to their respective device models, and then batches are stored.

[0057] The above functions are encapsulated as a Proxy module. This module does not need to be deployed separately. It can be introduced into the InfluxDB application that needs to be operated. When the business code calls, just call the provided API.

[0058] The operation method of data partitioning is as follows:

[0059] Data storage:

[0060] In step S1 ( Figure 3 ), the collection task calls the proxy encapsulation module program interface to directly store the data into the blocking queue, and the program returns immediately after storing the data in the queue.

[0061] In step S2 ( Figure 4 ), the proxy encapsulation module's warehousing task performs the actual data warehousing. First, it checks whether the storage data table corresponding to the data exists. If so, it directly obtains the Node of the corresponding node for operation. Otherwise, it uses the load balancing algorithm to find a suitable database. At the same time, the information about the data table being stored in this database is stored in the metadata table of the database ("measurement_map") and Redis, so that other business application instances can perceive the storage status of the data table.

[0062] 2. When querying data, the device being queried can know which data table should be queried, and then determine whether the data table already exists. If not, return empty directly. Otherwise, obtain the Node where the data table is located for query operations.

[0063] Summary: The InfluxDB operation proxy (Proxy) implemented in the present invention can store data in shards, which can ensure the efficiency of data writing when the amount of data is very large. At the same time, the producer-consumer model adopted further improves the efficiency of data storage; it also supports imperceptible horizontal expansion of the InfluxDB cluster.

[0064] Obviously, those skilled in the art should understand that the various modules of the cluster system of InfluxDB based on Redis or the various steps of the cluster system method of InfluxDB based on Redis of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A clustering method of InfluxDB based on Redis, characterized in that: When storing the collected device indicator data in InfluxDB, the data is sharded and stored in different InfluxDB instances. The method includes: (1) According to the characteristics of equipment indicator data in the industrial Internet field, the indicators of a device model are stored in a data table; (2) Each InfluxDB database has a metadata table that records the names of the data tables stored in this database; (3) When the application starts, it will load the metadata table information in each database according to the configured cluster information, obtain the storage status of the data table according to the metadata table, and store the currently read data table storage status in Redis; (4) When the device collects indicator data and puts it into the database, first query whether there is data table information corresponding to the device model in Redis. If it exists, write the data directly into the target table; (5) If the data table does not exist, an InfluxDB is selected according to the load balancing algorithm, and then the metadata table of the target database is updated and the data table information is written to indicate that the data table is stored in the target database; at the same time, the cache information in Redis is updated; the metadata is used to identify which database the data of the device model is stored in; (6) When querying device indicator data, first obtain the database instance corresponding to the device model data storage in the cache in Redis. If there is no database instance, return empty directly, otherwise query; (7) For data storage, the producer-consumer mode is adopted by default, and the collected indicator data is stored in a blocking queue. The storage task obtains batch data from the blocking queue and distributes it to the respective database nodes according to their respective device models, and then stores them in batches; (8) The above operations (3) to (7) are encapsulated into a Proxy module. The Proxy module does not need to be deployed separately. It can be introduced into the InfluxDB application that needs to be operated. When the business code operates the data, it can directly call the program interface provided by the Proxy module.

2. The clustering method of InfluxDB based on Redis according to claim 1, characterized in that: When querying data, because the indicators of the device model are stored in a data table, we know which data table should be queried according to the queried device, and then determine whether the data table already exists through the metadata table. If it does not exist, it returns empty directly. Otherwise, get the Node where the data table is located for query operations.

3. A system for implementing the clustering method of InfluxDB based on Redis as claimed in claim 1, characterized in that: Including node encapsulation module and agent encapsulation module; The node encapsulation module: encapsulates the InfluxDB database into a node, which contains the connection information of the InfluxDB database and the metadata table information of the database; encapsulates the operation of the data into a program interface according to the SDK provided by InfluxDB, and the parameter data format of the program interface is consistent with the requirements of the SDK; the node encapsulation module initializes the operating environment when the application starts, including establishing a network connection with InfluxDB and reading the metadata table to obtain the information of the current database storage table, and writing the information of the current database storage table into Redis; The proxy encapsulation module: encapsulates the operation of business data into a program interface according to the actual business situation. The data format of the program interface parameters is the format defined in the business system, and the program interface handles the conversion between the business system data format and the SDK data format; the proxy encapsulation module stores the current InfluxDB cluster node information, that is, multiple node information; the proxy encapsulation module contains a blocking queue for data entry to realize the temporary storage of the data entered, and a thread is used to consume the blocking queue data for entry; if the business system uses a distributed configuration center, the proxy encapsulation module can dynamically sense the configuration of the InfluxDB cluster and dynamically add the InfluxDB database.

4. The cluster system of InfluxDB based on Redis according to claim 3, characterized in that: The connection information refers to the IP address and port where the InfluxDB database is deployed.

5. The cluster system of InfluxDB based on Redis according to claim 3, characterized in that: The content in the metadata table is used to record which data tables are stored in the current database.

6. The cluster system of InfluxDB based on Redis according to claim 3, characterized in that: According to the characteristics of equipment indicator data in the industrial Internet field, the indicators of a device model are stored in a data table.

7. The cluster system of InfluxDB based on Redis according to claim 3, characterized in that: Each InfluxDB database has a metadata table.

8. The cluster system of InfluxDB based on Redis according to claim 3, characterized in that: The node encapsulation module encapsulates batch insertion and deletion operations of data into a program interface according to the SDK provided by InfluxDB.

9. The cluster system of InfluxDB based on Redis according to claim 3, characterized in that: The proxy encapsulation module encapsulates operations such as adding, deleting, modifying, and checking business data into a program interface.

10. The cluster system of InfluxDB based on Redis according to claim 3, characterized in that: When the application starts, it will load the metadata table information in each database according to the configured cluster information, obtain the storage status of the data table, and store the currently read data table storage status in Redis.

Images