Data processing method and device based on time sequence database and electronic equipment

Abstract

The invention provides a data processing method and device based on a time sequence database and electronic equipment, and relates to the technical field of databases. The method comprises the following steps: acquiring request information from a client; receiving the request information through a main server and analyzing the request information to obtain an analysis result; according to the analysis result and preset registration information, data read-write processing is carried out through a main cluster and a standby cluster, the registration information is used for representing data storage information corresponding to the main cluster and the standby cluster, and if any one of a main database and a standby database is in a network off-line state, the main database and the standby database are in a network off-line state. If yes, a processing request is sent to a proxy server corresponding to the other one of the main database and the standby database, and a temporary table is created. Based on an InfluxDb high-availability cluster, the high availability of an existing stand-alone version influxdb can be improved, and the functions of main and standby double writing and data synchronization after fault recovery are achieved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data processing method based on a time series database, a data processing device based on a time series database, an electronic device and a computer-readable storage medium. Background Art

[0002] In the field of database technology, various time series databases are used in the electric power industry, chemical industry, meteorological industry, geographic information and other types of real-time monitoring, inspection and analysis equipment to collect and generate data. The typical characteristics of these industrial data are: fast generation frequency, heavy dependence on collection time, large number of measurement points and large amount of information.

[0003] Currently, after the InfluxDb cluster version no longer develops source code, many existing technical solutions provide methods for data sharding. Using redis or zookeeper to manage the cluster can increase the performance of InfluxDb to a certain extent.

[0004] However, although the existing technology provides a cluster solution, it cannot achieve high availability. Because it does not provide redundancy, when a node in the cluster goes down, it will cause data loss on the failed node after the downtime, and the data of the shard on the node cannot be queried. There are also problems such as: after the downtime node fails, the data cannot be restored. Summary of the invention

[0005] The purpose of the present invention is to provide a data processing method, device and electronic device based on a time series database. Through the present invention, based on the cluster function of InfluxDb, the problem that after a single point of failure in the cluster, data query and storage of the shard at the node cannot be executed due to the failure is solved, thereby improving the high availability of the InfluxDb cluster.

[0006] In order to achieve the above object, an embodiment of the present invention provides a data processing method based on a time series database, which is applied to a database cluster. The time series database includes a master server and a master-slave cluster composed of multiple servers. The master-slave cluster includes a master database and a standby database. The method includes:

[0007] Get request information from the client;

[0008] The main server receives the request information and analyzes the request information to obtain an analysis result;

[0009] According to the analysis result and the pre-set registration information, data reading and writing processing is performed through the active-standby cluster, and the registration information is used to characterize the data storage information corresponding to the active-standby cluster. If the primary database and the standby database are both in an online network state, a processing request is sent to the proxy server corresponding to the primary database and the proxy server corresponding to the standby database respectively; if either the primary database or the standby database is in an offline network state, a processing request is sent to the proxy server corresponding to the other of the primary database and the standby database and a temporary table is created.

[0010] Specifically, before obtaining the request information from the client, the method further includes:

[0011] The registration information is obtained through ZooKeeper.

[0012] Specifically, before obtaining the registration information through zookeeper, the method further includes:

[0013] The registration information is generated by registering in Zookeeper according to the active and standby clusters through the proxy server.

[0014] Specifically, the client is a server client, the main server is InfluxDbCluster, and the request information is received and parsed by the main server to obtain a parsing result, including:

[0015] Receiving a query request statement from the server client through InfluxDbCluster;

[0016] The keywords of the query request statement are processed by a hash value calculation method to obtain the parsing result, and the parsing result is used to represent the corresponding position of the target data corresponding to the request information in the active-standby cluster.

[0017] Specifically, performing data reading and writing processing through the active / standby cluster according to the analysis result and the pre-set registration information includes:

[0018] If the primary database is in a network offline state or the target data fails to be written, a backup table is created in the standby database and the target data is written into the backup table. The primary database corresponds to the backup database one by one.

[0019] Specifically, before creating a backup table in the standby database and writing the target data into the backup table, the method further includes:

[0020] A scheduled task is started through a proxy server, and the scheduled task is used to confirm whether the backup table exists in the standby database at intervals of a time period.

[0021] Specifically, if the primary database is in a network offline state or the target data fails to be read or written, after creating a backup table in the standby database and writing the target data into the backup table, the method further includes:

[0022] The target data corresponding to the backup table is synchronized to the main database, and when the synchronization of the target data is completed, the backup table is deleted.

[0023] On the other hand, an embodiment of the present invention provides a data processing device based on a time series database, which is applied to a database cluster. The time series database includes a master server and a master-slave cluster composed of multiple servers. The master-slave cluster includes a master database and a standby database. The device includes:

[0024] A request information acquisition unit, used to acquire request information from a client;

[0025] A parsing result obtaining unit, used for receiving the request information through the main server and parsing the request information to obtain a parsing result;

[0026] A data processing unit is used to perform data reading and writing processing through the active-standby cluster according to the analysis result and pre-set registration information, wherein the registration information is used to characterize the data storage information corresponding to the active-standby cluster; if the primary database and the standby database are both in an online network state, a processing request is sent to a proxy server corresponding to the primary database and a proxy server corresponding to the standby database respectively; if either the primary database or the standby database is in an offline network state, a processing request is sent to a proxy server corresponding to the other of the primary database and the standby database and a temporary table is created.

[0027] In another aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0028] at least one processor;

[0029] a memory connected to the at least one processor;

[0030] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the aforementioned method by executing the instructions stored in the memory.

[0031] On the other hand, an embodiment of the present invention provides a computer-readable storage medium storing computer instructions, which, when executed on a computer, enable the computer to execute the aforementioned method.

[0032] The present invention is based on the InfluxDb high-availability cluster, which can improve the high availability of the existing InfluxDb cluster, realize the functions of master-slave dual writing and data synchronization after fault recovery, and restore the data lost during offline through the created temporary table after the server recovers from the offline state.

[0033] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0035] Figure 1 A flowchart of a data processing method based on a time series database according to an embodiment of the present application;

[0036] Figure 2 A schematic diagram of the structure of a data processing device based on a time series database provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0039] As used herein, the term "comprises" and any variation thereof, are intended to cover a non-exclusive inclusion.

[0040] Embodiments of the present application provide a data processing method, device, electronic device, and computer-readable storage medium based on a time series database.

[0041] The device can be integrated into a computer device, and the electronic device can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, or a personal computer (PC), etc. The server can be a single server or a server cluster composed of multiple servers.

[0042] In some embodiments, the device may also be integrated into multiple electronic devices. For example, the device may be integrated into multiple servers, and the method of the present application may be implemented by multiple servers.

[0043] In some embodiments, the server may also be implemented in the form of a terminal.

[0044] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0045] Example 1

[0046] An embodiment of the present invention provides a data processing method based on a time series database, which is applied to a database cluster. The time series database includes a master server and a master-slave cluster composed of multiple servers. The master-slave cluster includes a master database and a standby database.

[0047] like Figure 1 The specific process of the method includes steps 110 to 130:

[0048] 110. Get request information from the client.

[0049] In some embodiments of the present application, before step 110, the method further includes the following steps:

[0050] The registration information is obtained through ZooKeeper.

[0051] Specifically, before obtaining the registration information through Zookeeper, the method further includes:

[0052] The registration information is generated by registering in Zookeeper according to the active and standby clusters through the proxy server.

[0053] 120. Receive the request information through the main server and parse the request information to obtain a parsing result.

[0054] In some embodiments of the present application, the client is a server client, the master server is InfluxDbCluster, and step 120 includes the following steps:

[0055] 121. Receive a query request statement from the server client through InfluxDbCluster.

[0056] In some embodiments of the present application, different query request statements are used for different versions of influxdb. For example, when influxdb1.0, SQL statements are used, and when upgraded to influxdb2.0, Flux statements are used.

[0057] 122. Process the keywords of the query request statement by using a hash value calculation method to obtain the parsing result, where the parsing result is used to represent the corresponding position of the target data corresponding to the request information in the active-standby cluster.

[0058] 130. According to the analysis result and the pre-set registration information, data reading and writing are processed through the active-standby cluster, the registration information is used to characterize the data storage information corresponding to the active-standby cluster, and the registration information includes the address and port of the active-standby service; if the primary database and the standby database are both in an online network state, a processing request is sent to the proxy server corresponding to the primary database and the proxy server corresponding to the standby database respectively; if either the primary database or the standby database is in an offline network state, a processing request is sent to the proxy server corresponding to the other of the primary database and the standby database and a temporary table is created.

[0059] In some embodiments of the present application, step 130 includes the following specific process:

[0060] If the primary database is in a network offline state or the target data fails to be written, a backup table is created in the standby database and the target data is written into the backup table. The primary database corresponds to the backup database one by one.

[0061] Specifically, if the primary database is in a network offline state or the target data fails to be written, before creating a backup table in the standby database and writing the target data into the backup table, the method further includes the following specific processes:

[0062] A scheduled task is started through a proxy server, and the scheduled task is used to confirm whether the backup table exists in the standby database at intervals of a time period.

[0063] Specifically, if the primary database is in a network offline state or the target data fails to be read or written, after a backup table is created in the standby database and the target data is written into the backup table, the method further includes the following specific processes:

[0064] The target data corresponding to the backup table is synchronized to the main database, and when the synchronization of the target data is completed, the backup table is deleted.

[0065] Following the above-mentioned embodiment of the present application, the present embodiment is further described below through a specific implementation process:

[0066] Since the open source version of InfluxDb does not have clustering capabilities, in order to implement clustering, you need to customize a set of reverse proxy middleware and configure the proxy master and slave, and register with Zookeeper when the InfluxDb proxy master and slave are started.

[0067] The external system obtains the online status of the InfluxDb agent through Zookeeper to confirm the current request sending target. In the cluster, the InfluxDb agent parses the table name through the SQL request of the external system. Different tables are distributed to different InfluxDb databases in the cluster according to the hash value calculated by the table name. In this way, the load balancing of InfluxDb is realized, which includes the following:

[0068] (1) InfluxDbCluster:

[0069] It can be integrated in the App Server, responsible for receiving the request sent by the App Server, parsing the request, and calculating the master and standby group of the InfluxDb where the data is located through the hash value. Then, read and write operations are performed through the InfluxDb master and standby cluster registration information synchronized from Zookeeper.

[0070] (2) InfluxDbAgent:

[0071] It is mainly used to integrate InfluxDb and zookeeper, register InfluxDb information into zookeeper, and enable the InfluxDb cluster to have the characteristics of service discovery and health check.

[0072] (3) InfluxDbCluster obtains the health status of the master and standby InfluxDb through Zookeeper. If both the master and standby are normal, the request will be sent to the InfluxDb master and standby services at the same time, and the request will be returned successfully. If any of the master and standby InfluxDb services is offline, or any server fails to write, the request will only be sent to the other InfluxDb, and a temporary table will be created in the other InfluxDb, and the request will be returned successfully. If both the master and standby are offline, the request will be returned abnormally.

[0073] (4) When InfluxDb is offline or writing fails, a backup table will be created in another InfluxDb database, and the data that should have been written to the main database will be written to the corresponding backup table. The backup table and the actual table are in a one-to-one relationship.

[0074] Before this, InfluxDbAgent will start a scheduled task to check whether there is a backup table in the current database at regular intervals.

[0075] When a backup table exists and the other party of the master and backup is also online, the backup table content will be synchronized to another InfluxDb database. After confirming that a batch of data synchronization is completed, the backup data will be deleted. The synchronization frequency and the start interval of the scheduled task can be set.

[0076] In summary, the embodiments of the present invention have the following beneficial effects:

[0077] 1. High availability can reduce operation and maintenance costs, increase the stability of the InfluxDB cluster, ensure that the InfluxDB cluster can provide services to the outside world uninterruptedly, reduce losses caused by hardware, software, and human error, and increase concurrency. Zookeeper can configure multiple pairs of InfluxDB master and standby clusters;

[0078] 2. Load balancing: Through the influxdb master-slave cluster method, the system's data storage capacity can be increased and the influxdb cluster performance can be improved.

[0079] Example 2

[0080] The embodiment of the present invention and embodiment 1 belong to the same inventive concept. The embodiment of the present invention provides a data processing device based on a time series database, which is applied to a database cluster. The time series database includes a master server and a master-slave cluster composed of multiple servers. The master-slave cluster includes a master database and a standby database. Figure 2 As shown, the device comprises:

[0081] The request information acquisition unit 201 is used to acquire the request information from the client;

[0082] The parsing result obtaining unit 202 is used to receive the request information through the main server and parse the request information to obtain the parsing result;

[0083] The data processing unit 203 is used to perform data reading and writing processing through the active-standby cluster according to the analysis results and pre-set registration information, and the registration information is used to characterize the data storage information corresponding to the active-standby cluster. If the primary database and the standby database are both in an online state, a processing request is sent to the proxy server corresponding to the primary database and the proxy server corresponding to the standby database respectively; if either the primary database or the standby database is in an offline state, a processing request is sent to the proxy server corresponding to the other of the primary database and the standby database and a temporary table is created.

[0084] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0085] Example 3

[0086] The embodiment of the present invention and embodiments 1 and 2 all belong to the same inventive concept. The embodiment of the present invention provides an electronic device, the electronic device

[0087] It can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0088] In some embodiments, the apparatus may also be integrated into multiple devices. For example, the apparatus may be integrated into multiple servers, and the method of the present application may be implemented by multiple servers.

[0089] For example, Figure 3 As shown, it shows a schematic diagram of the structure of the device involved in the embodiment of the present application, specifically:

[0090] The device may include one or more processors 201 of processing cores, one or more storage media 202, a power supply 203, an input module 204, and a communication module 205. Those skilled in the art will appreciate that Figure 3 The device structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0091] The processor 201 is the control center of the device, and uses various interfaces and lines to connect various parts of the entire device. It executes various functions of the device and processes data by running or executing software programs and / or modules stored in the memory 202, and calling data stored in the memory 202. In some embodiments, the processor 201 may include one or more processing cores; in some embodiments, the processor 201 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 201.

[0092] The memory 202 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 202. The memory 202 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory 202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 202 may also include a memory controller to provide the processor 201 with access to the memory 202.

[0093] The device also includes a power supply 203 for supplying power to various components. In some embodiments, the power supply 203 can be logically connected to the processor 201 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 203 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0094] The device may further include an input module 204, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0095] The device may also include a communication module 205. In some embodiments, the communication module 205 may include a wireless module. The device may perform short-range wireless transmission through the wireless module of the communication module 205, thereby providing wireless broadband Internet access to the user. For example, the communication module 205 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0096] Although not shown, the device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 201 in the device will load the executable files corresponding to the processes of one or more application programs into the memory 202 according to the following instructions, and the processor 201 will run the application programs stored in the memory 202, thereby realizing various functions, as follows:

[0097] Get request information from the client;

[0098] The main server receives the request information and analyzes the request information to obtain an analysis result;

[0099] According to the analysis result and the pre-set registration information, data reading and writing processing is performed through the active-standby cluster, and the registration information is used to characterize the data storage information corresponding to the active-standby cluster. If the primary database and the standby database are both in an online network state, a processing request is sent to the proxy server corresponding to the primary database and the proxy server corresponding to the standby database respectively; if either the primary database or the standby database is in an offline network state, a processing request is sent to the proxy server corresponding to the other of the primary database and the standby database and a temporary table is created.

[0100] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0101] A person skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a storage medium and loaded and executed by a processor.

[0102] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0103] Get request information from the client;

[0104] The main server receives the request information and analyzes the request information to obtain an analysis result;

[0105] According to the analysis result and the pre-set registration information, data reading and writing processing is performed through the active-standby cluster, and the registration information is used to characterize the data storage information corresponding to the active-standby cluster. If the primary database and the standby database are both in an online network state, a processing request is sent to the proxy server corresponding to the primary database and the proxy server corresponding to the standby database respectively; if either the primary database or the standby database is in an offline network state, a processing request is sent to the proxy server corresponding to the other of the primary database and the standby database and a temporary table is created.

[0106] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0107] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0108] Since the instructions stored in the storage medium can execute the steps in any one of the methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0109] Some optional implementation modes of the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

[0110] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0111] In addition, various implementation modes of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed by the embodiments of the present invention.

Claims

1. A data processing method based on a time series database, characterized in that: Applied to a database cluster, the time series database includes a master server and a master-slave cluster consisting of multiple servers, the master-slave cluster includes a master database and a standby database, and the method includes: Get request information from the client; The main server receives the request information and analyzes the request information to obtain an analysis result; According to the analysis result and the pre-set registration information, data reading and writing processing is performed through the active-standby cluster, and the registration information is used to characterize the data storage information corresponding to the active-standby cluster. If the primary database and the standby database are both in an online network state, a processing request is sent to the proxy server corresponding to the primary database and the proxy server corresponding to the standby database respectively; if either the primary database or the standby database is in an offline network state, a processing request is sent to the proxy server corresponding to the other of the primary database and the standby database and a temporary table is created.

2. The method according to claim 1, characterized in that Before obtaining the request information from the client, the method further includes: The registration information is obtained through ZooKeeper.

3. The method according to claim 2, characterized in that Before obtaining the registration information through Zookeeper, the method further includes: The registration information is generated by registering in Zookeeper according to the active and standby clusters through the proxy server.

4. The method according to claim 1, characterized in that The client is a server client, the main server is InfluxDbCluster, and the request information is received and parsed by the main server to obtain a parsing result, including: Receiving a query request statement from the server client through InfluxDbCluster; The keywords of the query request statement are processed by a hash value calculation method to obtain the parsing result, and the parsing result is used to characterize the corresponding position of the target data corresponding to the request information in the active-standby cluster.

5. The method according to claim 1, characterized in that The data reading and writing processing is performed by the active / standby cluster according to the analysis result and the pre-set registration information, including: If the primary database is in a network offline state or the target data fails to be written, a backup table is created in the standby database and the target data is written into the backup table. The primary database corresponds to the backup database one by one.

6. The method according to claim 5, characterized in that Before creating a backup table in the standby database and writing the target data into the backup table, the method further includes: A scheduled task is started through a proxy server, and the scheduled task is used to confirm whether the backup table exists in the standby database at intervals of a time period.

7. The method according to claim 5, characterized in that After creating a backup table in the standby database and writing the target data into the backup table if the primary database is in a network offline state or the target data reading and writing fails, the method further includes: The target data corresponding to the backup table is synchronized to the main database, and when the synchronization of the target data is completed, the backup table is deleted.

8. A data processing device based on a time series database, characterized in that: Applied to a database cluster, the time series database includes a master server and a master-slave cluster composed of multiple servers, the master-slave cluster includes a master database and a standby database, and the device includes: A request information acquisition unit, used to acquire request information from a client; A parsing result obtaining unit, used for receiving the request information through the main server and parsing the request information to obtain a parsing result; A data processing unit is used to perform data reading and writing processing through the active-standby cluster according to the analysis result and pre-set registration information, wherein the registration information is used to characterize the data storage information corresponding to the active-standby cluster; if the primary database and the standby database are both in an online network state, a processing request is sent to a proxy server corresponding to the primary database and a proxy server corresponding to the standby database respectively; if either the primary database or the standby database is in an offline network state, a processing request is sent to a proxy server corresponding to the other of the primary database and the standby database and a temporary table is created.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method of any one of claims 1 to 7 by executing the instructions stored in the memory.

10. A computer-readable storage medium storing computer instructions, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 7.

Images