InfluxDB high-availability system and working method thereof

Abstract

The invention belongs to the field of computer data processing, and provides an InfluxDB high-availability system and a working method thereof in order to solve the problem that an existing InfluxDB lacks a mature high-availability scheme, the InfluxDB high-availability system comprises a client and an InfluxDB node, in the InfluxDB node, for a received data query request, the data query request is forwarded to a storage node assembly through a service interface assembly, and the storage node assembly is used for storing the data query request; the storage node component encapsulates the current query request into a query request supported by an InfluxDB query interface, performs InfluxDB data query, and returns a query result to the client; for a received data writing request, forwarding to-be-written data to the file storage component through the service interface component to perform data writing of a local file; writing the to-be-written data into the InfluxDB through the storage node component and returning to a writing state; the InfluxDB nodes are arranged in pairs, each pair of InfluxDB nodes is a main node and a standby node, and when data write-in exists, data synchronization of the main node and the standby node is carried out through a file synchronization assembly.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer data processing, and in particular relates to an InfluxDB high-availability system and a working method thereof. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] InfluxDB high availability means that a series of technologies and architectural designs are used to ensure that the InfluxDB database system can continue to provide services stably and continuously in the face of various failures and abnormal situations, thereby ensuring data availability and system reliability. InfluxDB is a time series database that does not support clustering. Currently, there are two InfluxDB high availability solutions, one is the Fork version of the official high availability solution InfluxDB-Relay, and the other is Influx-Proxy, an improved version of InfluxDB-Relay.

[0004] The forked version of InfluxDB-Relay is an HTTPServer to the outside world. After receiving a write request, it can support writing data to InfluxDB via HTTP or UDP, and can also support remote writing (HTTP) of Prometheus. However, InfluxDB-Relay does not support the query interface of InfluxDB, nor does it support remote reading of Prometheus. When the data write request reaches the InfluxDB-Relay layer, InfluxDB-Relay will initiate a write request to the InfluxDB of its configured backend. However, it does not solve the problem of data consistency. If writing to InfluxDB fails, it will simply return an error message, and the data that failed to be written will be lost in InfluxDB. Secondly, the InfluxDB-Proxy solution is mainly used for proxy and forwarding requests. It acts as an intermediate layer between the client and one or more InfluxDB instances, and can handle request forwarding in real time. It is transparent to the upper-level client, and the upper-level application can use it like a stand-alone InfluxDB. The proxy solution corresponds to a backend database for each write request, which makes it limited to one database. In addition, InfluxDB-Proxy requires additional configuration of KEYMAPS, which is mainly used in InfluxDB - Proxy to correctly map and adapt incoming requests to the backend InfluxDB service. KEYMAPS is essentially a configuration mechanism for handling possible differences between different client requests and backend databases. Write requests are randomly routed to the backend database, resulting in inconsistent storage of backend data of different databases, and each write is randomly routed, and the amount of written data cannot be controlled to be the same on each node, resulting in unbalanced data load.

[0005] In summary, the existing solutions lack a mature InfluxDB high availability solution, do not support query interfaces, cannot guarantee data consistency, and cannot quickly back up data. Summary of the invention

[0006] The embodiment of the present invention provides an InfluxDB high-availability system and a working method thereof to solve the problems that the traditional solution does not support the query interface, cannot guarantee data consistency, and cannot quickly back up data.

[0007] According to a first aspect of an embodiment of the present invention, an InfluxDB high availability system is provided, including: The client generates data queries or write requests based on user needs and sends them to the InfluxDB node. InfluxDB node, including the middle layer and InfluxDB, where: The middle layer includes a service interface component, a file storage component, a file synchronization component, and a storage node component. The middle layer is used to forward the received data query request to the storage node component through the service interface component, encapsulate the current query request as a query request supported by the query interface of InfluxDB through the storage node component, perform data query on InfluxDB, and return the query result to the client; for the received data write request, forward the data to be written to the file storage component through the service interface component to write the data to the local file, and at the same time, write the data to be written into InfluxDB through the storage node component, and return the write status; InfluxDB, which is used for data storage; The InfluxDB nodes are arranged in pairs, and each pair of InfluxDB nodes are mutually active and standby. When data is written, data synchronization of the active and standby nodes is performed through a file synchronization component.

[0008] Furthermore, data query or write request is generated based on user demand, wherein a unified destination address is used in the data query request or write request to query or write data, the operation status of the master and standby nodes is monitored through the Keepalive tool, and the virtual IP of the address is automatically switched to an InfluxDB node that can work normally.

[0009] Furthermore, the service interface component includes a data query interface, a Ping interface, a data write interface and a status notification interface. The service interface component specifically performs the following processing: calling the Ping interface through the virtual IP routing selection service in the Keepalive tool to determine whether the current InfluxDB node is running normally. If it is running normally, the current InfluxDB node is selected to perform data query or writing. If it is running abnormally, the backup node of the current InfluxDB node is selected to perform data query or writing; when querying data, by calling the data query interface, the data query request is forwarded to the storage node component through the data query interface, and the query result obtained by the storage node component is returned to the client; when writing data, the data to be written is forwarded to the file storage component through the data write interface to perform local file writing and write to the write channel of the storage node component.

[0010] Furthermore, the file storage component specifically performs the following processing procedure: when receiving newly written data, it queries the offset address when the latest synchronization is completed, uses the offset address as the data writing position, and writes the local file.

[0011] Furthermore, the file synchronization component specifically performs the following processing procedures: automatically read the local file of the current node, and read in the file data according to the offset address of the local cache when the latest synchronization is completed; establish a TCP connection between the current node and its corresponding standby node, and realize the synchronization of the local files of the two nodes based on TCP transmission.

[0012] Furthermore, after the service is initialized, if a certain InfluxDB node is selected as the master node, the file synchronization component specifically performs the following processing: automatically read the local file of the master node, read the file data according to the synchronization offset of the local cache, establish a TCP connection with another InfluxDB node, and synchronize the file; If an InfluxDB node is selected as a slave node, the file synchronization component specifically performs the following processing: creates a TCP client, establishes a connection between the TCP client and another InfluxDB node, and when it monitors that the other party has data synchronization, receives the data and writes it to the local file.

[0013] Furthermore, the storage node component specifically performs the following processing procedures: when receiving a data query request, encapsulating the request into a query request supported by the query interface of InfluxDB, querying InfluxDB, and encapsulating and returning the returned data; when receiving a data write request, parsing the data write request, assembling it into a corresponding SQL statement according to the influxdbsql syntax, writing the data into InfluxDB based on the SQL statement, and returning the write status.

[0014] According to a second aspect of an embodiment of the present invention, a working method of an InfluxDB high-availability system is provided, which is based on the above-mentioned InfluxDB high-availability system and includes: Based on the destination address in the data query or write request sent by the client, select the InfluxDB node that can normally execute the current request; In the selected InfluxDB node, for data query requests, the data query request is forwarded to the storage node component through the service interface component, and the current query request is encapsulated as a query request supported by the query interface of InfluxDB through the storage node component, and the InfluxDB data query is performed, and the query result is returned to the client; for data write requests, the data to be written is forwarded to the file storage component through the service interface component to write the data to the local file, and at the same time, the data to be written is written to InfluxDB through the storage node component, and the write status is returned; wherein, when there is data writing, the data synchronization of the master and standby nodes is performed through the file synchronization component.

[0015] Furthermore, a unified destination address is used in data query requests or write requests to query or write data, and the operating status of the active and standby nodes is monitored through the Keepalive tool, and the virtual IP of the address is automatically switched to an InfluxDB node that can work normally.

[0016] Furthermore, when a data write request selects a certain InfluxDB node, the local file of the current node is automatically read, and the file data is read in according to the offset address of the local cache when the latest synchronization is completed; a TCP connection is established between the current node and its corresponding standby node, and the synchronization of the local files of the two nodes is achieved based on TCP transmission.

[0017] One or more of the above technical solutions have the following beneficial effects: The invention provides an InfluxDB high-availability system and a working method thereof, comprising a client and an InfluxDB node. In the InfluxDB node, an intermediate layer is used for forwarding a received data query request to a storage node component through a service interface component, encapsulating the current query request as a query request supported by a query interface of InfluxDB through the storage node component, performing data query of InfluxDB, and returning the query result to the client; for a received data write request, forwarding the data to be written to a file storage component through the service interface component to write the data to a local file, and at the same time, writing the data to be written to InfluxDB through the storage node component, and returning the write status, providing a single entry point through a unified data read and write interface, simplifying user operations, and not caring about the back-end node status, and the external database read and write interface is fixed; at the same time, the InfluxDB nodes are arranged in pairs, and each pair of InfluxDB nodes are mutually active and standby. When data is written, the data synchronization of the active and standby nodes is performed through the file synchronization component, and the scheme adopts dual-machine automatic replication to realize real-time replication of data, ensure redundant backup of data, and through the local file synchronization mechanism, ensure that data is not lost even in the case of node failure.

[0018] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0020] Figure 1 This is a schematic diagram of the overall deployment of an InfluxDB high-availability system described in an embodiment of the present invention; Figure 2 This is a schematic diagram of overall deployment of an InfluxDB high-availability system to achieve data consistency between two nodes described in an embodiment of the present invention; Figure 3 This is a schematic diagram of the core modules of an InfluxDB high-availability system described in an embodiment of the present invention; Figure 4 A schematic diagram of the processing process of the file storage component described in an embodiment of the present invention; Figure 5 A schematic diagram of the processing process of the file synchronization component described in an embodiment of the present invention; Figure 6This is a schematic diagram of the data writing process described in an embodiment of the present invention; Figure 7 The figure is a schematic diagram of the data query process described in the embodiment of the present invention. DETAILED DESCRIPTION

[0021] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0022] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0023] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0024] Terminology explanation: InfluxDB: An open source time series database for efficient storage, retrieval, and analysis of time series data.

[0025] Synchronization Offset: refers to recording the current synchronization offset during data synchronization.

[0026] Keepalive tool: Mainly used to detect server status, prevent single point failures, and provide failover and load balancing functions.

[0027] SQL statement: A standard programming language specifically used to manage relational databases. Its core functions include data query, data manipulation, data definition, and data control.

[0028] TCP: A connection-oriented, reliable, byte-stream-based transport layer communication protocol defined by IETF RFC 793.

[0029] In one or more embodiments, Figure 1-Figure 7 As shown, an InfluxDB high availability system is provided, including: The client generates data queries or write requests based on user needs and sends them to the InfluxDB node. InfluxDB node, including the middle layer and InfluxDB, where: The middle layer includes a service interface component, a file storage component, a file synchronization component, and a storage node component. The middle layer is used to forward the received data query request to the storage node component through the service interface component, encapsulate the current query request as a query request supported by the query interface of InfluxDB through the storage node component, perform data query on InfluxDB, and return the query result to the client; for the received data write request, forward the data to be written to the file storage component through the service interface component to write the data to the local file, and at the same time, write the data to be written into InfluxDB through the storage node component, and return the write status; InfluxDB, which is used for data storage; The InfluxDB nodes are arranged in pairs, and each pair of InfluxDB nodes are mutually active and standby. When data is written, data synchronization of the active and standby nodes is performed through a file synchronization component.

[0030] InfluxDB high availability system uses local file automatic replication service to achieve data consistency. Figure 2 As shown in the figure, after the data is written to the local file of the InfluxDB node, it will automatically synchronize to another InfluxDB node to ensure the consistency of the local file data of the InfluxDB node. Each InfluxDB node deploys the influxdb-relay service to maintain a local file. The user's read and write requests first request the influxdb-relay service, and the influxdb-relay service will write the data to the local file and store it in the database at the same time. The influxdb-relay service includes components such as Httpserver, FileBackend, FileSync, and InfluxBackend, which are easy to manage and expand.

[0031] The automatic replication service synchronizes local files between InfluxDB nodes to ensure file data consistency, thereby achieving data consistency between two InfluxDB nodes.

[0032] In a specific implementation, data query or write request is generated based on user demand, wherein a unified destination address is used in the data query request or write request to query or write data, the operation status of the master and standby nodes is monitored through the Keepalive tool, and the virtual IP of the address is automatically switched to an InfluxDB node that can work normally.

[0033] Specifically, the solution described in this embodiment uses keepalive to achieve automatic switching of VIP (virtual IP). Users are not aware of the backend influxdb node and use a unified address to query and write data. After receiving the message, the service automatically routes and directly sends the message to influxdb. When the database node goes offline, the online node is automatically selected to ensure the high availability of the service. The overall deployment diagram is as follows: Figure 1 shown.

[0034] This embodiment adopts the VIP automatic switching method and uses the Keepalive technology to realize automatic failover of the virtual IP to ensure service continuity.

[0035] In a specific implementation, the service interface component includes a data query interface, a Ping interface, a data write interface and a status notification interface. The service interface component specifically performs the following processing: calling the Ping interface through the virtual IP routing selection service in the Keepalive tool to determine whether the current InfluxDB node is running normally. If it is running normally, the current node is selected to perform data query or writing. If it is running abnormally, the backup node of the current node is selected for data query or writing; when querying data, by calling the data query interface, the data query request is forwarded to the storage node component through the data query interface, and the query result obtained by the storage node component is returned to the client; when writing data, the data to be written is forwarded to the file storage component through the data write interface to perform local file writing and write to the write channel of the storage node component.

[0036] In a specific implementation, the file storage component specifically performs the following processing: when receiving newly written data, it queries the offset address when the latest synchronization is completed, uses the offset address as the data writing position, and writes the local file.

[0037] Specifically, the file storage component, namely the FileBackend component, provides file writing and file management functions, such as Figure 4 As shown, the specific processing process includes the following: 1) File writing: ① Listen to the initialization notification channel. When the file synchronization initialization is completed, query the synchronized offset and write the synchronized offset into the file.

[0038] ② After initialization is completed, when new data is received, the latest synchronized offset is queried and written to the file. At the same time, the data is written to the data writing channel of the storage node component (i.e., influxdbBackend component).

[0039] 2) File management: ① During initialization, it detects whether there is a file. If there is no file, it creates a file. The file is set to a fixed size. When writing data, the file size is determined. If it exceeds the set size value, a new file is automatically created.

[0040] ② Scan all files in the path regularly, and delete those files whose file size has been filled and whose file contents have been fully synchronized.

[0041] In a specific implementation, the file synchronization component specifically performs the following processing: when a data write request selects a certain node, it automatically reads the local file of the current node, and reads in the file data according to the offset address when the latest synchronization is completed in the local cache; a TCP connection is established between the current node and its corresponding standby node, and the synchronization of the local files of the two nodes is achieved based on TCP transmission.

[0042] Specifically, the file synchronization component, FleSync, provides file synchronization. After the service is initialized, if the InfluxDB node is selected as the master node, it automatically reads the local file of the node, reads the file data according to the synchronization offset of the local cache, establishes a TCP connection with another InfluxDB node, and synchronizes the file. After the service is initialized, if the InfluxDB node is selected as the slave node, it creates a TCP client and establishes a connection with another InfluxDB node. When it detects that the other party has data synchronization, it receives the data and writes it to the local file.

[0043] Implement TCP server and TCP client, such as Figure 5 shown.

[0044] 1) TCP server: ① Receive the master status signal, start the server, and stop the client.

[0045] ② Receive slave synchronization data and write it to the file and channel.

[0046] ③ Receive slave synchronization request, read files, synchronize unsynchronized data, and update synchronization offset. Send synchronization initialization completion signal. (If the connection fails after timeout, the initialization is considered completed).

[0047] ④ Synchronize the data received in the synchronization channel.

[0048] 2) TCP client: ① Send a request to the master; after the connection is established, read the most recent data, get the offset, scan the file to synchronize the unsynchronized data; request the master to synchronize the data.

[0049] ② Receive synchronous data and write it to the library channel.

[0050] In a specific implementation, the storage node component specifically performs the following processing procedures: when a data query request is received, the request is encapsulated as a query request supported by the query interface of InfluxDB, the InfluxDB query is performed, and the returned data is encapsulated and returned; when a data write request is received, the data write request is parsed, assembled into a corresponding data write SQL statement according to the influxdbsql syntax, the data is written to InfluxDB based on the SQL statement, and the write status is returned.

[0051] In one or more implementations, this embodiment provides a working method of an InfluxDB high availability system, which is based on the above-mentioned InfluxDB high availability system and includes: Based on the destination address in the data query or write request sent by the client, select the InfluxDB node that can normally execute the current request; In the selected InfluxDB node, for data query requests, the data query request is forwarded to the storage node component through the service interface component, and the current query request is encapsulated as a query request supported by the query interface of InfluxDB through the storage node component, and the InfluxDB data query is performed, and the query result is returned to the client; for data write requests, the data to be written is forwarded to the file storage component through the service interface component to write the data to the local file, and at the same time, the data to be written is written to InfluxDB through the storage node component, and the write status is returned; wherein, when there is data writing, the data synchronization of the master and standby nodes is performed through the file synchronization component.

[0052] In the specific implementation, a unified destination address is used in data query requests or write requests to query or write data. The operating status of the active and standby nodes is monitored through the Keepalive tool, and the virtual IP of the address is automatically switched to an InfluxDB node that can work normally.

[0053] In the specific implementation, when a data write request selects a certain InfluxDB node, the local file of the current node is automatically read, and the file data is read in according to the offset address when the latest synchronization is completed in the local cache; a TCP connection is established between the current node and its corresponding standby node, and the synchronization of the local files of the two nodes is achieved based on TCP transmission.

[0054] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in the present embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An InfluxDB high availability system, characterized in that: include: The client generates data queries or write requests based on user needs and sends them to the InfluxDB node. InfluxDB node, including the middle layer and InfluxDB, where: The middle layer includes a service interface component, a file storage component, a file synchronization component, and a storage node component. The middle layer is used to forward the received data query request to the storage node component through the service interface component, encapsulate the current query request as a query request supported by the query interface of InfluxDB through the storage node component, perform data query on InfluxDB, and return the query result to the client; for the received data write request, forward the data to be written to the file storage component through the service interface component to write the data to the local file, and at the same time, write the data to be written into InfluxDB through the storage node component, and return the write status; InfluxDB, which is used for data storage; The InfluxDB nodes are arranged in pairs, and each pair of InfluxDB nodes are mutually active and standby. When data is written, data synchronization of the active and standby nodes is performed through a file synchronization component.

2. An InfluxDB high availability system as claimed in claim 1, characterized in that: The method generates data query or write request based on user demand, wherein a unified destination address is used in the data query request or write request to query or write data, the operation status of the master and standby nodes is monitored through the Keepalive tool, and the virtual IP of the address is automatically switched to an InfluxDB node that can work normally.

3. The InfluxDB high availability system according to claim 1, characterized in that: The service interface component includes a data query interface, a Ping interface, a data write interface and a status notification interface. The service interface component specifically performs the following processing: calling the Ping interface through the virtual IP routing selection service in the Keepalive tool to determine whether the current InfluxDB node is running normally. If it is running normally, the current InfluxDB node is selected to perform data query or write. If it is running abnormally, the backup node of the current InfluxDB node is selected to perform data query or write; when querying data, by calling the data query interface, the data query request is forwarded to the storage node component through the data query interface, and the query result obtained by the storage node component is returned to the client; when writing data, the data to be written is forwarded to the file storage component through the data write interface to perform local file writing and write to the write channel of the storage node component.

4. The InfluxDB high availability system according to claim 1, characterized in that: The file storage component specifically performs the following processing process: when receiving newly written data, it queries the offset address when the latest synchronization is completed, uses the offset address as the data writing position, and writes the local file.

5. The InfluxDB high availability system according to claim 1, characterized in that: The file synchronization component specifically performs the following processing procedures: automatically read the local file of the current node, and read the file data according to the offset address of the local cache when the latest synchronization is completed; establish a TCP connection between the current node and its corresponding standby node, and realize the synchronization of the local files of the two nodes based on TCP transmission.

6. An InfluxDB high availability system as claimed in claim 5, characterized in that: After the service is initialized, if a certain InfluxDB node is selected as the master node, the file synchronization component specifically performs the following processing: automatically read the local file of the master node, read the file data according to the synchronization offset of the local cache, establish a TCP connection with another InfluxDB node, and synchronize the file; If an InfluxDB node is selected as a slave node, the file synchronization component specifically performs the following processing: creates a TCP client, establishes a connection between the TCP client and another InfluxDB node, and when it monitors that the other party has data synchronization, receives the data and writes it to the local file.

7. The InfluxDB high availability system according to claim 1, characterized in that: The storage node component specifically performs the following processing: when receiving a data query request, encapsulates the request into a query request supported by the query interface of InfluxDB, performs a query on InfluxDB, and encapsulates and returns the returned data; When a data write request is received, the data write request is parsed, assembled into a corresponding SQL statement according to the influxdb sql syntax, the data is written to InfluxDB based on the SQL statement, and the write status is returned.

8. A working method of an InfluxDB high availability system, characterized in that: It is based on an InfluxDB high availability system according to any one of claims 1 to 7, including: Based on the destination address in the data query or write request sent by the client, select the InfluxDB node that can normally execute the current request; In the selected InfluxDB node, for data query requests, the data query request is forwarded to the storage node component through the service interface component, and the current query request is encapsulated as a query request supported by the query interface of InfluxDB through the storage node component, and the InfluxDB data query is performed, and the query result is returned to the client; for data write requests, the data to be written is forwarded to the file storage component through the service interface component to write the data to the local file, and at the same time, the data to be written is written to InfluxDB through the storage node component, and the write status is returned; wherein, when there is data writing, the data synchronization of the master and standby nodes is performed through the file synchronization component.

9. A method for operating an InfluxDB high-availability system as claimed in claim 8, characterized in that: A unified destination address is used in data query requests or write requests to query or write data. The Keepalive tool is used to monitor the operating status of the active and standby nodes, and the virtual IP of the address is automatically switched to an InfluxDB node that can work normally.

10. A method for operating an InfluxDB high availability system according to claim 8, characterized in that: When a data write request selects a certain InfluxDB node, the local file of the current node is automatically read, and the file data is read in according to the offset address of the local cache when the latest synchronization is completed; a TCP connection is established between the current node and its corresponding standby node, and the synchronization of the local files of the two nodes is achieved based on TCP transmission.

Images