Time series database system and data processing method and device

Abstract

The present invention discloses a time series database system and a data processing method and device, and relates to the field of computer technology. The time series database system includes: a deployment service module and a management service module; and the deployment service module is used to: receive a request to deploy an InfluxDB cluster, obtain configuration information of the InfluxDB cluster to be deployed according to the request to deploy the InfluxDB cluster, and deploy at least one InfluxDB cluster according to the configuration information, wherein each InfluxDB cluster includes at least one InfluxDB node; the management service module is used to: manage the working information and parameter information of at least one InfluxDB cluster. The time series database system can realize multi-tenant mode and on-demand allocation, and can also facilitate intuitive operation and use by users, providing users with a better experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a time series database system and a data processing method and device. Background Art

[0002] InfluxDB is a maintenance-free, stable, reliable, and elastically scalable time series database product that can provide automated time series data collection, compressed storage, SQL-like query, multi-dimensional aggregation computing, and data visualization analysis capabilities. At present, the open source version of InfluxDB is a stand-alone version. In order to solve the single point failure problem, the following InfluxDB high availability solution is proposed. The specific implementation is: three InfluxDB nodes can be started, and the distributed consistency protocol is used to ensure the reliability of node state switching. The data log is synchronously copied from the master node to the two standby nodes. When at least two nodes in the cluster are written successfully, the data write is completed and submitted.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: (1) It is impossible to select the version of Influxdb and the free configuration, resulting in the inability to allocate on demand; (2) There is a lack of a visual interface, and the configuration parameters cannot be modified in real time, resulting in a poor user experience; (3) When writing data, at least two nodes must be written successfully before it can be submitted, resulting in low writing efficiency; (4) The sequential voting strategy of the consistency protocol has a great impact on multi-table transactions, resulting in potential performance and stability risks. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a time series database system and a data processing method and device, which can realize multi-tenant mode and on-demand allocation, facilitate user intuitive operation and use, and provide users with a better experience.

[0005] To achieve the above objective, according to a first aspect of an embodiment of the present invention, a time series database system is provided.

[0006] The time series database system of an embodiment of the present invention includes: a deployment service module and a management service module; and the deployment service module is used to: receive a request to deploy an InfluxDB cluster, obtain configuration information of the InfluxDB cluster to be deployed according to the request to deploy the InfluxDB cluster, and deploy at least one InfluxDB cluster according to the configuration information, wherein each InfluxDB cluster includes at least one InfluxDB node; the management service module is used to: manage the working information and parameter information of the at least one InfluxDB cluster.

[0007] Optionally, the configuration information includes at least one of the following options: the number of InfluxDB clusters to be deployed, the number of InfluxDB nodes included in each InfluxDB cluster to be deployed, version information corresponding to the InfluxDB nodes, and hardware information corresponding to the InfluxDB nodes.

[0008] Optionally, the deployment service module is also used to: deploy the InfluxDB service corresponding to each InfluxDB node on the physical machine according to the hardware information corresponding to each InfluxDB node and the hardware information of the physical machine, so as to obtain each InfluxDB node.

[0009] Optionally, the deployment service module is further used to: expand and / or shrink the at least one InfluxDB cluster according to the expansion and / or shrink request.

[0010] Optionally, the management service module is also used to: monitor, for each InfluxDB cluster, working information of at least one InfluxDB node contained in each InfluxDB cluster; wherein the working information includes at least one of the following options: the working status of the InfluxDB node, the write request information of the InfluxDB node, and the memory usage of the InfluxDB node.

[0011] Optionally, the management service module is also used to: provide a management interface for each InfluxDB cluster to facilitate controlling parameter information of at least one InfluxDB node contained in each InfluxDB cluster; wherein the parameter information includes at least one of the following options: the working status of the InfluxDB node, the service port corresponding to the InfluxDB node, the data storage path corresponding to the InfluxDB node, and the metadata directory corresponding to the InfluxDB node.

[0012] Optionally, the management service module is further used to: display configuration information of the InfluxDB cluster to be deployed.

[0013] Optionally, the system further includes: a load balancing module; wherein the load balancing module is used to: poll the working status of the at least one InfluxDB node so as to distribute the data processing request to the at least one InfluxDB node according to the working status.

[0014] Optionally, the system also includes: a proxy service module and a daemon module; and the daemon module is used to: monitor the working status of the at least one InfluxDB node, and write the physical machine information corresponding to the InfluxDB node whose working status is activated into the proxy service module; the load balancing module is also used to: check the physical machine information written in the proxy service module, and distribute the data processing request to the InfluxDB node whose working status is activated according to the written physical machine information.

[0015] Optionally, the daemon module is further used to: restart the InfluxDB node where the exception occurs, and / or switch to a standby InfluxDB node corresponding to the InfluxDB node where the exception occurs.

[0016] Optionally, the number of the proxy service modules is at least one, and the proxy service module has a one-to-one relationship and / or a one-to-many relationship with the at least one InfluxDB node; and the number of the daemon process modules is at least one, and the daemon process module has a one-to-one relationship and / or a one-to-many relationship with the at least one InfluxDB node.

[0017] Optionally, the system further includes: a file storage service module; and the InfluxDB nodes included in each InfluxDB cluster mount the same file storage service module.

[0018] To achieve the above objective, according to a second aspect of an embodiment of the present invention, a data processing method is provided.

[0019] The data processing method of an embodiment of the present invention includes: receiving a data processing request, obtaining a target InfluxDB cluster corresponding to the data processing request; polling the working status of at least one InfluxDB node included in the target InfluxDB cluster, and determining at least one optional InfluxDB node; selecting a target InfluxDB node from the at least one optional InfluxDB node, and distributing the data processing request to the target InfluxDB node, so that the target InfluxDB node processes the data processing request.

[0020] Optionally, before polling the working status of at least one InfluxDB node included in the target InfluxDB cluster, the method further includes: using a daemon process module to monitor the working status of the at least one InfluxDB node, and writing physical machine information corresponding to the InfluxDB node whose working status is an activated state into the proxy service module.

[0021] Optionally, polling the working status of at least one InfluxDB node contained in the target InfluxDB cluster to determine at least one optional InfluxDB node includes: checking the physical machine information written in the proxy service module; obtaining the at least one optional InfluxDB node based on the written physical machine information; wherein the optional InfluxDB node is an InfluxDB node whose working status is an activated state.

[0022] Optionally, the method further includes: when an abnormality occurs in the target InfluxDB node, restarting the target InfluxDB node, and / or switching the data processing request to a standby InfluxDB node corresponding to the target InfluxDB node.

[0023] Optionally, selecting a target InfluxDB node from the at least one optional InfluxDB node includes: randomly selecting a node from the at least one optional InfluxDB node as the target InfluxDB node; and, calculating a node score of the at least one optional InfluxDB node based on working information of the at least one optional InfluxDB node, and selecting the optional InfluxDB node with the highest score as the target InfluxDB node.

[0024] To achieve the above objective, according to a third aspect of an embodiment of the present invention, a data processing device is provided.

[0025] The data processing device of an embodiment of the present invention includes: an acquisition module, used to receive a data processing request, and acquire a target InfluxDB cluster corresponding to the data processing request; a determination module, used to poll the working status of at least one InfluxDB node included in the target InfluxDB cluster, and determine at least one optional InfluxDB node; a processing module, used to select a target InfluxDB node from the at least one optional InfluxDB node, and distribute the data processing request to the target InfluxDB node, so that the target InfluxDB node processes the data processing request.

[0026] Optionally, the determination module is further used to: use the daemon process module to monitor the working status of the at least one InfluxDB node, and write the physical machine information corresponding to the InfluxDB node whose working status is an activated state into the proxy service module.

[0027] Optionally, the determination module is also used to: check the physical machine information written in the proxy service module; obtain the at least one optional InfluxDB node according to the written physical machine information; wherein the optional InfluxDB node is an InfluxDB node whose working status is an activated state.

[0028] Optionally, the device also includes an exception handling module, which is used to: restart the target InfluxDB node when an exception occurs in the target InfluxDB node, and / or switch the data processing request to a standby InfluxDB node corresponding to the target InfluxDB node.

[0029] Optionally, the processing module is also used to: randomly select a node from the at least one optional InfluxDB node as the target InfluxDB node; and, based on the working information of the at least one optional InfluxDB node, calculate the node score of the at least one optional InfluxDB node, and select the optional InfluxDB node with the highest score as the target InfluxDB node.

[0030] To achieve the above objective, according to a fourth aspect of an embodiment of the present invention, an electronic device is provided.

[0031] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to the embodiment of the present invention.

[0032] To achieve the above objective, according to a fifth aspect of an embodiment of the present invention, a computer-readable medium is provided.

[0033] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, the data processing method according to the embodiment of the present invention is implemented.

[0034] One embodiment of the above invention has the following advantages or beneficial effects: in the time series database system of the embodiment of the present invention, the deployment service module can deploy at least one InfluxDB cluster according to the configuration information, and different InfluxDB clusters can be isolated from each other in terms of network and resources, so as to realize multi-tenant mode and on-demand allocation. In addition, the management service module manages the working information and parameter information of the InfluxDB cluster, avoiding the problem that the configuration parameters cannot be corrected in real time in the prior art, facilitating the user to intuitively operate and use the time series database system, and providing a better experience for the user. In addition, the configuration information includes the number of InfluxDB clusters to be deployed and the number of InfluxDB nodes contained in each InfluxDB cluster to be deployed, so as to realize the deployment of a single-machine version or a multi-node mode cluster according to the data scale and business requirements, and has scalability. In addition, the configuration information includes the version information corresponding to the InfluxDB node and the hardware information corresponding to the InfluxDB node, so that the version of InfluxDB can be selected and the physical machine can be freely configured, so as to further realize on-demand allocation.

[0035] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0037] Figure 1 is a schematic diagram of the structure of a time series database system according to an embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of main steps of a data processing method based on a time series database system according to an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of the main process of a data processing method based on a time series database system according to an embodiment of the present invention;

[0040] Figure 4 is a schematic diagram of main modules of a data processing device based on a time series database system according to an embodiment of the present invention;

[0041] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0042] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0044] InfluxDB has the following features: (1) High-performance data storage tailored for time series data; (2) Simple and efficient write and query interfaces; (3) Tailor-made SQL-like query language for easy query of aggregated data; (4) Allows for indexing of tags to achieve fast and efficient queries; (5) Data retention policy that can effectively invalidate old data automatically. Therefore, InfluxDB is an excellent choice for processing time series data. However, the open source version currently provided is a stand-alone version, and the InfluxDB high-availability solution has the problems of being unable to allocate on demand, unable to modify configuration parameters in real time, low write efficiency, and potential performance and stability risks. Therefore, an embodiment of the present invention provides a time series database system that can automatically deploy an InfluxDB cluster, integrate data processing and operation and maintenance, and improve user experience.

[0045] Figure 1 FIG. 1 is a schematic diagram of the structure of a time series database system according to an embodiment of the present invention. Figure 1 As shown, the time series database system 100 may include: a deployment service module 101 and a management service module 102 .

[0046] Among them, the deployment service module 101 can be used to: receive a request to deploy an InfluxDB cluster, obtain the configuration information of the InfluxDB cluster to be deployed according to the request to deploy the InfluxDB cluster, and deploy at least one InfluxDB cluster according to the configuration information. It should be noted that each InfluxDB cluster can contain at least one InfluxDB node. The deployment service module refers to a module that can provide deployment services. Preferably, the Sahara service can be selected as the deployment service module. The Sahara service can quickly deploy a cluster within a few minutes, and also supports elastic expansion of nodes. It can easily increase or decrease computing nodes on demand to achieve elastic data computing services.

[0047] The configuration information may include at least one of the following options: (1) the number of InfluxDB clusters to be deployed, that is, the number of InfluxDB clusters to be deployed; (2) the number of InfluxDB nodes contained in each InfluxDB cluster to be deployed, the number of nodes contained in each InfluxDB cluster to be deployed as specified in the configuration information; (3) the version information corresponding to the InfluxDB node, such as version 1.7.9 or version 2.0; (4) the hardware information corresponding to the InfluxDB node, that is, the hardware information such as CPU, memory and hard disk space required by each node. After receiving a specific request to deploy an InfluxDB cluster, the deployment service module 101 can obtain the configuration information of the InfluxDB cluster to be deployed according to the received request, and then deploy at least one InfluxDB cluster according to the obtained configuration information.

[0048] In the embodiment of the present invention, the deployment service module 101 can also be used to: deploy the InfluxDB service corresponding to each InfluxDB node on the physical machine according to the hardware information corresponding to each InfluxDB node and the hardware information of the physical machine to obtain each InfluxDB node. Deploying an InfluxDB cluster is actually deploying the InfluxDB service on a physical machine (which can be understood as a computer host) to obtain an InfluxDB node. The hardware information corresponding to a certain InfluxDB node is obtained from the configuration information. During the deployment process, the physical machine that satisfies the hardware information corresponding to the InfluxDB node can be obtained according to the hardware information of the physical machine, and then the service corresponding to the node is deployed on the obtained physical machine, so that the InfluxDB node can be obtained. In other words, the InfluxDB node can be understood as a physical machine on which the InfluxDB service is deployed. The deployment service module 101 can deploy the same InfluxDB service on multiple physical machines, or it can deploy multiple InfluxDB services on the same physical machine, so that the InfluxDB cluster can be deployed using a cloud host, that is, the time series database system of the embodiment of the present invention supports cloud features.

[0049] In addition, in an embodiment of the present invention, the deployment service module 101 can also be used to: expand and / or shrink at least one InfluxDB cluster according to expansion and / or shrink requests. Assume that the deployment service module 101 deploys an InfluxDB cluster according to the configuration information, and the InfluxDB cluster includes 5 InfluxDB nodes. The deployment service module 101 can expand the InfluxDB cluster, such as adding 1 InfluxDB node, or expanding the hardware information corresponding to an InfluxDB node among the 5 InfluxDB nodes. The deployment service module 101 can shrink the InfluxDB cluster, such as reducing 1 InfluxDB node, or shrinking the hardware information corresponding to an InfluxDB node among the 5 InfluxDB nodes. The deployment service module 101 can expand and / or shrink, which further reflects the scalability of the time series database system.

[0050] Deployment service module 101 is a key part of deploying the InfluxDB cluster. Before deploying the InfluxDB cluster, you need to create the deployment service module 101 first. If you deploy the Saraha service first, it should be noted that the deployment of the Saraha service requires two physical machines to form a master-slave relationship to avoid the problem of being unable to create a cluster due to a physical machine crash. Then, you can use the deployed Sahara service to create a time series database cluster. When creating it, you need to set the configuration information first, then click Create Cluster, and then wait for about ten minutes to complete the creation of the cluster, realizing automatic deployment of the cluster.

[0051] In the time series database system of the embodiment of the present invention, the deployment service module can deploy at least one InfluxDB cluster according to the configuration information, and different InfluxDB clusters can be isolated from each other in terms of network and resources to realize a multi-tenant mode. In addition, the configuration information includes the number of InfluxDB clusters to be deployed and the number of InfluxDB nodes contained in each InfluxDB cluster to be deployed, so as to realize the deployment of a single-machine version or a multi-node mode cluster according to the data scale and business requirements, and has scalability. In addition, the configuration information includes the version information corresponding to the InfluxDB node and the hardware information corresponding to the InfluxDB node, so that the version of InfluxDB can be selected and the physical machine can be freely configured to realize on-demand allocation.

[0052] The management service module 102 may be used to manage the working information and parameter information of at least one InfluxDB cluster. The management service module 102 may be an Ambari service, which may provide a unified management and monitoring platform to monitor the working information and parameter information of at least one InfluxDB node included in each InfluxDB cluster.

[0053] The working information may include at least one of the following options: the working state of the InfluxDB node, the write request information of the InfluxDB node, and the memory usage of the InfluxDB node. The working state may include: the active state (i.e., the state in which the service can be provided) and the dormant state (i.e., the state in which the service cannot be provided); the write request information may include: the rate of the write request and the number of the write request; the memory usage may reflect the memory usage of the InfluxDB node.

[0054] In an embodiment of the present invention, the management service module 102 may also be used to provide a management interface for each InfluxDB cluster to facilitate control of parameter information of at least one InfluxDB node included in each InfluxDB cluster. The parameter information may include at least one of the following options: the working status of the InfluxDB node, the service port corresponding to the InfluxDB node, the data storage path corresponding to the InfluxDB node, and the metadata directory corresponding to the InfluxDB node.

[0055] The management service module 102 can provide a management interface, so that users can intuitively operate and use the time series database system. On the management interface, the working information of each InfluxDB node can be displayed, that is, the working status of each InfluxDB node, the write request information of the InfluxDB node, and the memory usage of the InfluxDB node can be displayed. In order to facilitate real-time viewing of the number of write requests of the InfluxDB node, the management service module 102 can also configure icons to observe the usage over a period of time. In addition, the management service module 102 can also set an alarm mechanism, such as when the number of write requests of the InfluxDB node reaches a certain threshold, it can actively remind the user.

[0056] In addition, the parameter information of each InfluxDB node can be displayed on the management interface. In this way, users can control the start and stop status of each InfluxDB node, such as switching an active InfluxDB node to a dormant state. Of course, users can also change the service port, data storage path, and metadata directory of each InfluxDB node through the management interface.

[0057] In an embodiment of the present invention, the management service module 102 can also be used to: display the configuration information of the InfluxDB cluster to be deployed. That is, the management service module 102 can be used to display the configuration information of the InfluxDB cluster to be deployed, so that the user can adjust the configuration information through the management service module 102, and then the deployment service module 101 can deploy the InfluxDB cluster according to the adjusted configuration information. In addition, the expansion and / or reduction request can also be displayed through the management service module 102, so that after the user confirms, the deployment service module 101 can expand and / or reduce the capacity according to the expansion and / or reduction request.

[0058] In the embodiment of the present invention, after the cluster is deployed using the deployment service module 101, the management service module 102 provides a management interface login method, so that parameter information and configuration information can be adjusted in real time, and work information can be monitored, giving users a better user experience. In addition, the management service module 102 can provide a backend login method, so that the cluster's command line mode can be entered from the backend to perform operations such as creating a library table.

[0059] like Figure 1 As shown, the time series database system 100 may further include: a load balancing module 103, a proxy service module 104, a daemon module 105 and a file storage service module 106. The load balancing module 103, the proxy service module 104, the daemon module 105 and the file storage service module 106 may play an auxiliary role in the process of processing data requests by the InfluxDB cluster deployed in the time series database cluster, and provide high availability characteristics of the time series database cluster.

[0060] Among them, the load balancing module 103 can be used to: poll the working status of at least one InfluxDB node, so as to distribute the data processing request to at least one InfluxDB node according to the working status. Specifically, the load balancing module 103 can periodically query the working status of at least one InfluxDB node, obtain the InfluxDB node in the activated state, so as to distribute the data processing request to the InfluxDB node in the activated state in time, reduce the processing time of the data processing request, and improve the processing efficiency of the request. In addition, the use of the load balancing module 103 to distribute the data processing request avoids the sequential voting strategy of the consistency protocol in the prior art.

[0061] In the embodiment of the present invention, the load balancing module 103 can distribute the data processing request to the node together with the proxy service module 104 and the daemon module 105. Specifically, the daemon module 105 can be used to: monitor the working status of at least one InfluxDB node, and write the physical machine information corresponding to the InfluxDB node whose working status is activated into the proxy service module 104. The load balancing module 103 can also be used to: view the physical machine information written in the proxy service module 104, and distribute the data processing request to the InfluxDB node whose working status is activated according to the written physical machine information. It can be seen that the proxy service module 104 plays the role of guiding the load balancing module 103 to access. Among them, the proxy service module 104 can select the Nginx (i.e., a free, open source, high-performance HTTP server and reverse proxy server) service. The role of Nginx is to proxy the real InfluxDB service. When the proxy InfluxDB service is detected to be invalid, Nginx can modify its own configuration to proxy another InfluxDB.

[0062] In the embodiment of the present invention, the daemon module 105 can also be used to restart the abnormal InfluxDB node, and / or switch to the standby InfluxDB node corresponding to the abnormal InfluxDB node. That is, the daemon module 104 can regularly detect the service status of the InfluxDB node. When the service provided by the InfluxDB node is abnormal or the InfluxDB node is down, the daemon module 104 is responsible for restarting the InfluxDB node or switching it to the standby node.

[0063] The file storage service module 106 can be used to provide the underlying data storage service for the InfluxDB node. In an embodiment of the present invention, the InfluxDB nodes contained in each InfluxDB cluster mount the same file storage service module, that is, the storage data directory and pre-write directory of the InfluxDB nodes contained in each InfluxDB cluster are set on the same shared file storage, so that high data availability can be achieved. It can be seen that the number of file storage service modules 106 can be one or more. Generally, an InfluxDB cluster corresponds to a business line, so all InfluxDB nodes contained in an InfluxDB cluster can be mounted to a file storage service module 106, and accordingly, all InfluxDB nodes contained in another InfluxDB cluster can be mounted to another file storage service module 106. In other words, it is ensured that the InfluxDB nodes contained in each InfluxDB cluster are mounted to the same file storage service module 106. Of course, if there is only one file storage service module 106, all InfluxDB nodes contained in the deployed InfluxDB cluster can be mounted to the file storage service module 106. Since multiple InfluxDB nodes can be mounted to the same file storage service module 106, the problem in the prior art that at least two nodes must be written successfully before submission can be solved, thereby improving data writing efficiency. In addition, the file storage service can have a built-in multi-copy mechanism, which automatically generates corresponding copies after writing data to ensure the integrity and reliability of the data.

[0064] In addition, the number of proxy service modules 104 is at least one, and the proxy service module 104 and at least one InfluxDB node are in a one-to-one relationship and / or a one-to-many relationship; and the number of daemon modules 105 is at least one, and the daemon module 105 and at least one InfluxDB node are in a one-to-one relationship and / or a one-to-many relationship. Specifically, in an embodiment of the present invention, one InfluxDB node may correspond to one proxy service module 104, or one proxy service module 104 may correspond to multiple InfluxDB nodes. Similarly, one InfluxDB node may correspond to a daemon module 105, or one daemon module 105 may correspond to multiple InfluxDB nodes.

[0065] In the embodiment of the present invention, the load balancing module 103, the proxy service module 104, the daemon module 105 and the file storage service module 106 included in the time series database system 100 are all for high availability escort. These modules are automatically created in the process of deploying the cluster. The load balancing module 103 provides a unified entrance. After the load balancing module 103, the request will be distributed to the proxy service module 104. According to the information of the proxy service module 104, the InfluxDB node in the activated state is selected, so as to access the Influxdb service in real operation and process the request. The daemon module 105 can check the service status of the InfluxDB node regularly. If the service of the InfluxDB node is down or stops abnormally, the daemon module 105 will automatically pull up the InfluxDB node. If it fails to pull up multiple times, the InfluxDB node will be automatically switched to the standby node to start. In addition, the InfluxDB nodes contained in the same InfluxDB cluster will mount the same file storage service module 106, so as to provide basic conditions for node switching. It should be noted that only the InfluxDB node in operation can read and write the file storage service module 106, thereby ensuring data consistency. In addition, the file storage service module 106 can be stored as multiple redundancies, thereby having extremely high service availability and data reliability.

[0066] In the time series database system of the embodiment of the present invention, the deployment service module can deploy at least one InfluxDB cluster according to the configuration information, and different InfluxDB clusters can be isolated from each other in terms of network and resources, so as to realize multi-tenant mode and on-demand allocation. In addition, the management service module manages the working information and parameter information of the InfluxDB cluster, avoiding the problem that the configuration parameters cannot be corrected in real time in the prior art, facilitating the user to intuitively operate and use the time series database system, and providing a better experience for the user. In addition, the configuration information includes the number of InfluxDB clusters to be deployed and the number of InfluxDB nodes contained in each InfluxDB cluster to be deployed, so as to realize the deployment of a single-machine version or a multi-node mode cluster according to the data scale and business requirements, and has scalability. In addition, the configuration information includes the version information corresponding to the InfluxDB node and the hardware information corresponding to the InfluxDB node, so that the version of InfluxDB can be selected and the physical machine can be freely configured, so as to further realize on-demand allocation.

[0067] For ease of understanding, a method for performing data processing using at least one InfluxDB cluster deployed by a time series database system, that is, a data processing method based on a time series data system, is described in detail below. Figure 2 FIG. 1 is a schematic diagram of the main steps of a data processing method based on a time series database system according to an embodiment of the present invention. Figure 2 As shown, the main steps of the data processing method based on the time series database system may include:

[0068] Step S201, receiving a data processing request, and obtaining a target InfluxDB cluster corresponding to the data processing request;

[0069] Step S202, polling the working status of at least one InfluxDB node included in the target InfluxDB cluster, and determining at least one optional InfluxDB node;

[0070] Step S203: Select a target InfluxDB node from at least one optional InfluxDB node, and distribute the data processing request to the target InfluxDB node, so that the target InfluxDB node processes the data processing request.

[0071] The time series database system of the embodiment of the present invention can deploy at least one InfluxDB cluster. Considering that different InfluxDB clusters correspond to different business lines, after receiving a data processing request, the request can be analyzed to obtain the target InfluxDB cluster corresponding to the request, that is, the business line corresponding to the request is analyzed, and the InfluxDB cluster corresponding to the business line is determined to be the target InfluxDB cluster. Of course, if the number of InfluxDB clusters in the embodiment of the present invention is 1, then the InfluxDB cluster can be directly determined as the target InfluxDB cluster. In addition, in addition to being distinguished by business line, the InfluxDB clusters in the embodiment of the present invention can also be distinguished by other dimensional indicators, such as data type, which is not limited by the embodiment of the present invention.

[0072] After obtaining the target InfluxDB cluster, you can poll at least one InfluxDB node included in the target InfluxDB cluster, that is, query the working status of all InfluxDB nodes included in the InfluxDB cluster. According to the working status of the node, select an optional InfluxDB node from the at least one InfluxDB node included in the target InfluxDB cluster. The optional InfluxDB node is defined as an InfluxDB node whose working status is activated.

[0073] Finally, a target InfluxDB node may be selected from at least one optional InfluxDB node, and then the received data processing request may be distributed to the target InfluxDB node so that the target InfluxDB node processes the data processing request.

[0074] In an embodiment of the present invention, before polling the working status of at least one InfluxDB node included in the target InfluxDB cluster, the data processing method may also include: using a daemon process module to monitor the working status of at least one InfluxDB node, and writing the physical machine information corresponding to the InfluxDB node whose working status is an activated state into the proxy service module.

[0075] The functions of the daemon module and the proxy service module have been explained above. During the data processing process, the daemon module can be used to periodically obtain the working status of at least one InfluxDB node contained in the target InfluxDB cluster and select the active node. Then the physical machine information corresponding to the active InfluxDB node is written to the proxy service module. In this way, the optional InfluxDB node can be directly determined by querying the proxy service module, thereby reducing data processing time and improving data processing efficiency.

[0076] Therefore, after writing the physical machine information corresponding to the InfluxDB node whose working state is activated into the proxy service module, step S202 may include: checking the physical machine information written in the proxy service module; and obtaining at least one optional InfluxDB node according to the written physical machine information.

[0077] In addition, in an embodiment of the present invention, the data processing method may further include: restarting the target InfluxDB node when an abnormality occurs in the target InfluxDB node, and / or switching the data processing request to a standby InfluxDB node corresponding to the target InfluxDB node. In an embodiment of the present invention, the abnormality is handled by restarting or switching, thereby ensuring that data processing can be processed normally and improving the user experience.

[0078] In an embodiment of the present invention, the following two methods can be selected to select a target InfluxDB node from at least one optional InfluxDB node: Method 1: randomly select a node from at least one optional InfluxDB node as the target InfluxDB node; Method 2: calculate the node score of at least one optional InfluxDB node based on the working information of at least one optional InfluxDB node, and select the optional InfluxDB node with the highest score as the target InfluxDB node. The working information may include: working status, write request rate, number of write requests, and memory usage.

[0079] Since the optional InfluxDB node is an activated InfluxDB node, that is, the optional InfluxDB node is an InfluxDB node that is working normally, in method 1, a node can be randomly selected from the optional InfluxDB nodes as the target InfluxDB node. In method 2, the working status, write request rate, number of write requests, and memory usage of each optional InfluxDB node are comprehensively considered. Each optional InfluxDB node is scored, and the best optional InfluxDB node is determined as the target InfluxDB node.

[0080] Figure 3 FIG. 1 is a schematic diagram of the main flow of a data processing method based on a time series database system according to an embodiment of the present invention. Figure 3 As shown, the main process of the data processing method based on the time series database system may include:

[0081] Step S301, receiving a data processing request, and obtaining a target InfluxDB cluster corresponding to the data processing request;

[0082] Step S302, using a daemon module to monitor the working status of at least one InfluxDB node included in the target InfluxDB cluster;

[0083] Step S303, writing the physical machine information corresponding to the InfluxDB node whose working status is activated into the proxy service module;

[0084] Step S304, checking the physical machine information written in the proxy service module;

[0085] Step S305, obtaining at least one optional InfluxDB node according to the written physical machine information;

[0086] Step S306, randomly selecting a node from at least one optional InfluxDB node as a target InfluxDB node;

[0087] Step S307, calculating a node score of at least one optional InfluxDB node according to the working information of at least one optional InfluxDB node;

[0088] Step S308, selecting the optional InfluxDB node with the highest score as the target InfluxDB node;

[0089] Step S309: distribute the data processing request to the target InfluxDB node so that the target InfluxDB node processes the data processing request.

[0090] It should be noted that steps S302 and S303 are used to monitor the working status of the InfluxDB node, and write the physical machine information of the activated InfluxDB node into the proxy service module, so that in the process of data processing requests, the optional InfluxDB node can be obtained only by querying the proxy service module, thereby reducing the data processing time. In addition, steps S302 and S303 are executed regularly, that is, the optional InfluxDB node is placed in the physical machine at regular intervals. In addition, step S306 is a way to select the target InfluxDB node, and steps S307 and S308 are another way to select the target InfluxDB node. When executing a data processing request, you can choose according to actual needs.

[0091] Figure 4 Schematic diagram of main modules of a data processing device based on a time series database system according to an embodiment of the present invention. Figure 4 As shown, the data processing device 400 based on the time series database system may include: an acquisition module 401 , a determination module 402 and a processing module 403 .

[0092] Among them, the acquisition module 401 can be used to: receive a data processing request, and obtain a target InfluxDB cluster corresponding to the data processing request; the determination module 402 can be used to: poll the working status of at least one InfluxDB node contained in the target InfluxDB cluster, and determine at least one optional InfluxDB node; the processing module 403 can be used to: select a target InfluxDB node from at least one optional InfluxDB node, and distribute the data processing request to the target InfluxDB node, so that the target InfluxDB node processes the data processing request.

[0093] In the embodiment of the present invention, the determination module 402 may also be used to: monitor the working status of at least one InfluxDB node using the daemon module, and write the physical machine information corresponding to the InfluxDB node whose working status is activated into the proxy service module.

[0094] In an embodiment of the present invention, the determination module 402 can also be used to: view the physical machine information written in the proxy service module; obtain at least one optional InfluxDB node according to the written physical machine information; wherein the optional InfluxDB node is an InfluxDB node whose working state is an activated state.

[0095] In the embodiment of the present invention, the data processing device 400 based on the time series database system may further include: an exception handling module (not shown in the figure). The exception handling module may be used to: restart the target InfluxDB node when an exception occurs in the target InfluxDB node, and / or switch the data processing request to the standby InfluxDB node corresponding to the target InfluxDB node.

[0096] In the embodiment of the present invention, the processing module 403 may also be used to: randomly select a node from at least one optional InfluxDB node as a target InfluxDB node; and, based on working information of at least one optional InfluxDB node, calculate a node score of at least one optional InfluxDB node, and select the optional InfluxDB node with the highest score as the target InfluxDB node.

[0097] Figure 5 An exemplary system architecture 500 is shown to which a data processing method based on a time series database system or a data processing device based on a time series database system according to an embodiment of the present invention can be applied.

[0098] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0099] The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various applications can be installed on the terminal devices 501, 502, 503, such as applications that provide data processing services.

[0100] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0101] Server 505 may be a server that provides various services, such as a background management server (only an example) that provides support for data processing requests sent by users using terminal devices 501, 502, and 503. The background management server may analyze and process the received data processing requests and feed back the processing results to the terminal devices.

[0102] It should be noted that the data processing method based on the time series database system provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data processing device based on the time series database system is generally set on the server 505 .

[0103] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0104] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0105] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0106] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0107] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0108] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0109] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, a determination module, and a processing module. The names of these modules do not constitute a limitation on the modules themselves in some cases. For example, the acquisition module may also be described as "a module that receives a data processing request and obtains a target InfluxDB cluster corresponding to the data processing request".

[0111] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: receiving a data processing request, obtaining a target InfluxDB cluster corresponding to the data processing request; polling the working status of at least one InfluxDB node included in the target InfluxDB cluster, and determining at least one optional InfluxDB node; selecting a target InfluxDB node from at least one optional InfluxDB node, and distributing the data processing request to the target InfluxDB node, so that the target InfluxDB node processes the data processing request.

[0112] According to the technical solution of the embodiment of the present invention, the deployment service module of the time series database system can deploy at least one InfluxDB cluster according to the configuration information, and different InfluxDB clusters can be isolated from each other in terms of network and resources, so as to realize multi-tenant mode and on-demand allocation. In addition, the management service module manages the working information and parameter information of the InfluxDB cluster, avoiding the problem that the configuration parameters cannot be corrected in real time in the prior art, facilitating the user's intuitive operation and use of the time series database system, and providing a better experience for the user. In addition, the configuration information includes the number of InfluxDB clusters to be deployed and the number of InfluxDB nodes contained in each InfluxDB cluster to be deployed, so as to realize the deployment of a single-machine version or a multi-node mode cluster according to the data scale and business requirements, and has scalability. In addition, the configuration information includes the version information corresponding to the InfluxDB node and the hardware information corresponding to the InfluxDB node, so that the version of InfluxDB can be selected and the physical machine can be freely configured, so as to further realize on-demand allocation.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A time series database system, characterized in that: The system comprises: a deployment service module and a management service module; and The deployment service module is used to: receive a request to deploy an InfluxDB cluster, obtain configuration information of the InfluxDB cluster to be deployed according to the request to deploy the InfluxDB cluster, and deploy at least one InfluxDB cluster according to the configuration information, wherein each InfluxDB cluster includes at least one InfluxDB node; The management service module is used to: manage the working information and parameter information of the at least one InfluxDB cluster; The system further includes: a load balancing module; wherein the load balancing module is used to: poll the working status of the at least one InfluxDB node, so as to distribute the data processing request to the at least one InfluxDB node according to the working status; The system further comprises: an agent service module and a daemon process module; and The daemon process module is used to: monitor the working status of the at least one InfluxDB node, and write the physical machine information corresponding to the InfluxDB node whose working status is an activated state into the proxy service module; The load balancing module is also used to: view the physical machine information written in the proxy service module, and obtain at least one optional InfluxDB node according to the written physical machine information; calculate the node score of at least one optional InfluxDB node according to the working information of at least one optional InfluxDB node; select the optional InfluxDB node with the highest score as the target InfluxDB node; distribute the data processing request to the target InfluxDB node so that the target InfluxDB node processes the data processing request; wherein the proxy service module and the at least one InfluxDB node are in a one-to-many relationship, and the daemon process module and the at least one InfluxDB node are in a one-to-many relationship; The management service module is also used to provide a management interface for each InfluxDB cluster, and display the working information of each InfluxDB node on the management interface, including the write request information of the InfluxDB node and the memory usage of the InfluxDB node. The usage over a period of time is displayed by configuring icons, and an alarm mechanism can also be set to actively remind when the number of write requests of the InfluxDB node reaches a certain threshold.

2. The system according to claim 1, characterized in that The configuration information includes at least one of the following options: the number of InfluxDB clusters to be deployed, the number of InfluxDB nodes included in each InfluxDB cluster to be deployed, version information corresponding to the InfluxDB nodes, and hardware information corresponding to the InfluxDB nodes.

3. The system according to claim 2, characterized in that The deployment service module is also used for: According to the hardware information corresponding to each InfluxDB node and the hardware information of the physical machine, the InfluxDB service corresponding to each InfluxDB node is deployed on the physical machine to obtain each InfluxDB node.

4. The system according to claim 3, characterized in that The deployment service module is further used to: expand and / or shrink the at least one InfluxDB cluster according to the expansion and / or shrink request.

5. The system according to claim 1, characterized in that The management service module is also used for: For each InfluxDB cluster, monitor the working information of at least one InfluxDB node included in each InfluxDB cluster; wherein, The working information includes at least one of the following options: working status of the InfluxDB node, write request information of the InfluxDB node, and memory usage of the InfluxDB node.

6. The system according to claim 5, characterized in that The management service module is also used for: Providing a management interface for each InfluxDB cluster to facilitate control of parameter information of at least one InfluxDB node included in each InfluxDB cluster; wherein, The parameter information includes at least one of the following options: a working status of the InfluxDB node, a service port corresponding to the InfluxDB node, a data storage path corresponding to the InfluxDB node, and a metadata directory corresponding to the InfluxDB node.

7. The system according to claim 1, characterized in that The management service module is also used to: display the configuration information of the InfluxDB cluster to be deployed.

8. The system according to claim 1, characterized in that The daemon module is also used to: restart the InfluxDB node where the exception occurs, and / or switch to a standby InfluxDB node corresponding to the InfluxDB node where the exception occurs.

9. The system according to any one of claims 1 to 8, characterized in that: The system further includes: a file storage service module; and The InfluxDB nodes included in each InfluxDB cluster mount the same file storage service module.

10. A data processing method, characterized in that: include: Receive a data processing request, and obtain a target InfluxDB cluster corresponding to the data processing request; Polling the working status of at least one InfluxDB node included in the target InfluxDB cluster to determine at least one optional InfluxDB node; Selecting a target InfluxDB node from the at least one optional InfluxDB node, and distributing the data processing request to the target InfluxDB node, so that the target InfluxDB node processes the data processing request; Before polling the working status of at least one InfluxDB node included in the target InfluxDB cluster, the method further includes: Using the daemon module to monitor the working status of the at least one InfluxDB node, and writing the physical machine information corresponding to the InfluxDB node whose working status is an activated state into the proxy service module; The polling the working status of at least one InfluxDB node included in the target InfluxDB cluster to determine at least one optional InfluxDB node includes: Check the physical machine information written in the proxy service module; According to the written physical machine information, obtaining the at least one optional InfluxDB node; wherein the optional InfluxDB node is an InfluxDB node whose working state is an activated state; According to the working information of the at least one optional InfluxDB node, the node score of the at least one optional InfluxDB node is calculated, and the optional InfluxDB node with the highest score is selected as the target InfluxDB node; wherein the proxy service module and the at least one InfluxDB node are in a one-to-many relationship, and the daemon process module and the at least one InfluxDB node are in a one-to-many relationship; It also includes: displaying the working information of each InfluxDB node, including the write request information of the InfluxDB node and the memory usage of the InfluxDB node, displaying the usage over a period of time through the configuration icon, and actively reminding when the number of write requests of the InfluxDB node reaches a certain threshold through the set alarm mechanism.

11. The method according to claim 10, characterized in that The method further comprises: In the event that an abnormality occurs in the target InfluxDB node, the target InfluxDB node is restarted, and / or the data processing request is switched to a standby InfluxDB node corresponding to the target InfluxDB node.

12. The method according to claim 10, characterized in that The selecting a target InfluxDB node from the at least one optional InfluxDB node comprises: A node is randomly selected from the at least one optional InfluxDB node as the target InfluxDB node.

13. A data processing device, characterized in that: include: An acquisition module, used to receive a data processing request and acquire a target InfluxDB cluster corresponding to the data processing request; A determination module, configured to poll the working status of at least one InfluxDB node included in the target InfluxDB cluster, and determine at least one optional InfluxDB node; a processing module, configured to select a target InfluxDB node from the at least one optional InfluxDB node, and distribute the data processing request to the target InfluxDB node, so that the target InfluxDB node processes the data processing request; The determination module is specifically used to monitor the working status of the at least one InfluxDB node by using the daemon process module, write the physical machine information corresponding to the InfluxDB node whose working status is activated into the proxy service module; check the physical machine information written in the proxy service module; obtain the at least one optional InfluxDB node according to the written physical machine information; wherein the optional InfluxDB node is the InfluxDB node whose working status is activated; The processing module is specifically configured to calculate the node score of the at least one optional InfluxDB node according to the working information of the at least one optional InfluxDB node, and select the optional InfluxDB node with the highest score as the target InfluxDB node; wherein the proxy service module and the at least one InfluxDB node are in a one-to-many relationship, and the daemon process module and the at least one InfluxDB node are in a one-to-many relationship; The determination module is specifically used to: display the working information of each InfluxDB node, including the write request information of the InfluxDB node and the memory usage of the InfluxDB node, display the usage for a period of time by configuring icons, and actively remind when the number of write requests of the InfluxDB node reaches a certain threshold through the set alarm mechanism.

14. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 10 to 12.

15. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 10 to 12 is implemented.

Images