Influxdb high availability system and working method thereof

Abstract

The application belongs to the field of computer data processing, and aims at solving the problem that the existing InfluxDB lacks mature high-availability solutions, and provides an InfluxDB high-availability system and a working method thereof, which comprises a client and an InfluxDB node, wherein, for a received data query request, the InfluxDB node forwards the data query request to a storage node component through a service interface component, the storage node component encapsulates the current query request into a query request supported by a query interface of InfluxDB, performs data query of InfluxDB, and returns the query result to the client; for a received data write request, the InfluxDB node forwards the data to be written to a file storage component through the service interface component to perform data writing of a local file; the InfluxDB node writes the data to be written into InfluxDB through the storage node component and returns a write state; the InfluxDB node is arranged in pairs, each pair of InfluxDB nodes is a master and a backup, and when there is data writing, the master and the backup perform data synchronization through a file synchronization component.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing technology, and in particular relates to an InfluxDB high-availability system and its working method. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] InfluxDB high availability refers to a series of technologies and architectural designs that ensure the InfluxDB database system can continue to provide services stably and continuously when facing various failures and anomalies, guaranteeing data availability and system reliability. InfluxDB, as an instant-series database, does not support clustering. Currently, there are two InfluxDB high availability solutions: one is a fork version of the officially provided high availability solution InfluxDB-Relay, and the other is an improved version of InfluxDB-Relay, Influx-Proxy.

[0004] The forked version of the InfluxDB-Relay solution acts as an HTTP server. Upon receiving a write request, it can write data to InfluxDB via HTTP or UDP, and also supports remote writes (HTTP) to Prometheus. However, InfluxDB-Relay does not support InfluxDB's query interface or Prometheus's remote reads. When a data write request reaches the InfluxDB-Relay layer, it initiates a write request to its configured backend InfluxDB. However, it does not solve the data consistency problem. If the write to InfluxDB fails, it simply returns an error message, and the failed data is lost in InfluxDB.

[0005] Secondly, the InfluxDB-Proxy solution is primarily used for proxying and forwarding requests. It acts as an intermediary layer between the client and one or more InfluxDB instances, handling request forwarding in real time. It is transparent to upper-layer clients, allowing applications to use it as if it were a single-machine InfluxDB instance. However, the proxy solution assigns one backend database to each write request, limiting it to a single database. Furthermore, InfluxDB-Proxy requires additional KEYMAPS configuration. In InfluxDB-Proxy, KEYMAPS is mainly used to correctly map and adapt incoming requests to the backend InfluxDB service. KEYMAPS is essentially a configuration mechanism to handle potential differences between different client requests and backend databases. Randomly routing write requests to the backend database leads to inconsistent data storage across different backend databases. Moreover, since each write is randomly routed, it's impossible to control the amount of data written across all nodes, resulting in unbalanced data load.

[0006] In summary, existing solutions lack a mature high-availability solution for InfluxDB, do not support query interfaces, cannot guarantee data consistency, and cannot quickly back up data. Summary of the Invention

[0007] This invention provides an InfluxDB high-availability system and its working method to solve the problems of traditional solutions not supporting query interfaces, failing to guarantee data consistency, and being unable to quickly back up data.

[0008] According to a first aspect provided by an embodiment of the present invention, an InfluxDB high-availability system is provided, comprising:

[0009] The client is used to generate data query or write requests based on user needs and distribute them to the InfluxDB nodes;

[0010] InfluxDB nodes, including the middleware layer and InfluxDB, include:

[0011] The middle layer includes a service interface component, a file storage component, a file synchronization component, and a storage node component. For received data query requests, the middle layer forwards the data query request to the storage node component via the service interface component. The storage node component then encapsulates the current query request into a query request supported by the InfluxDB query interface, performs the InfluxDB data query, and returns the query results to the client. For received data write requests, the middle layer forwards the data to be written to the file storage component via the service interface component for local file writing. Simultaneously, the storage node component writes the data to be written to InfluxDB and returns the write status.

[0012] InfluxDB is used for data storage;

[0013] The InfluxDB nodes are configured in pairs, with each pair serving as both primary and backup nodes. When data is written, the data is synchronized between the primary and backup nodes through a file synchronization component.

[0014] Furthermore, the process of generating data query or write requests based on user needs involves using a unified destination address for both the query and write requests. The Keepalive tool is used to monitor the operational status of the primary and backup nodes and automatically switches the virtual IP address to a working InfluxDB node.

[0015] Furthermore, the service interface component includes a data query interface, a Ping interface, a data write interface, and a status notification interface. Specifically, the service interface component performs the following processing steps: It uses the virtual IP routing service in the Keepalive tool to call the Ping interface to determine if the current InfluxDB node is running normally. If it is running normally, it selects the current InfluxDB node to perform data query or write; if it is running abnormally, it selects a backup node of the current InfluxDB node for data query or write. During data query, it calls the data query interface, forwards the data query request to the storage node component, and returns the query results obtained by the storage node component to the client. During data write, it uses the data write interface to forward the data to be written to the file storage component to perform local file write and write to the write channel of the storage node component.

[0016] Furthermore, the file storage component specifically performs the following process: when newly written data is received, it queries the offset address at the time of the latest synchronization and uses the offset address as the data writing position to write the local file.

[0017] Furthermore, the file synchronization component specifically performs the following processing steps: automatically reads the local file of the current node, and reads the file data according to the offset address of the latest synchronization completed in the local cache; establishes a TCP connection between the current node and its corresponding backup node, and realizes the synchronization of local files between the two nodes based on TCP transmission.

[0018] Furthermore, after service initialization, if an InfluxDB node is selected as the master node, the file synchronization component will perform the following process: automatically read the files on the master node, read the file data according to the local cached synchronization offset, establish a TCP connection with another InfluxDB node, and synchronize the files.

[0019] If an InfluxDB node is selected as a slave node, the file synchronization component performs the following process: it creates a TCP client, establishes a connection between the TCP client and another InfluxDB node, and receives the data and writes it to a local file when it detects that the other party is synchronizing data.

[0020] Furthermore, the storage node component specifically performs the following processing steps: when a data query request is received, the request is encapsulated into a query request supported by the InfluxDB query interface, an InfluxDB query is performed, and the returned data is encapsulated and returned; when a data write request is received, the data write request is parsed, assembled into a corresponding SQL statement according to the InfluxDB SQL syntax, the data is written to InfluxDB based on the SQL statement, and the write status is returned.

[0021] According to a second aspect of the present invention, a method for operating an InfluxDB high-availability system is provided, which is based on the above-described InfluxDB high-availability system and includes:

[0022] Based on the destination address in the data query or write request sent by the client, select the InfluxDB node that can execute the current request normally;

[0023] In the selected InfluxDB node, for data query requests, the service interface component forwards the data query request to the storage node component. The storage node component then encapsulates the current query request into a query request supported by the InfluxDB query interface, performs the InfluxDB data query, and returns the query results to the client. For data write requests, the service interface component forwards the data to be written to the file storage component for local file writing. Simultaneously, the storage node component writes the data to be written to InfluxDB and returns the write status. When data is being written, the file synchronization component performs data synchronization between the primary and backup nodes.

[0024] Furthermore, a unified destination address is used for both data query and write requests. The Keepalive tool is used to monitor the operation of the primary and backup nodes and automatically switch the virtual IP address to the InfluxDB node that is working properly.

[0025] Furthermore, when a data write request selects a specific InfluxDB node, it automatically reads the local file of the current node and reads the file data according to the offset address of the latest synchronization completed in the local cache; it establishes a TCP connection between the current node and its corresponding backup node, and realizes the synchronization of local files between the two nodes based on TCP transmission.

[0026] The above one or more technical solutions have the following beneficial effects:

[0027] This invention provides a high-availability InfluxDB system and its operating method, including a client and InfluxDB nodes. Within the InfluxDB nodes, a middleware layer forwards received data query requests to a storage node component via a service interface component. The storage node component encapsulates the current query request into a query request supported by the InfluxDB query interface, performs the InfluxDB data query, and returns the query results to the client. For received data write requests, the service interface component forwards the data to be written to a file storage component for local file writing. Simultaneously, the storage node component writes the data to be written to InfluxDB and returns the write status. A unified data read / write interface provides a single entry point, simplifying user operations without requiring concern for backend node status; the external database read / write interfaces are fixed. Furthermore, InfluxDB nodes are configured in pairs, with each pair acting as primary and backup for each other. When data is being written, a file synchronization component synchronizes the data between the primary and backup nodes. The solution employs automatic dual-machine replication to achieve real-time data replication, ensuring redundant backup. The local file synchronization mechanism also guarantees no data loss even in the event of node failure.

[0028] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0030] Figure 1 This is a schematic diagram illustrating the overall deployment of an InfluxDB high-availability system as described in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram illustrating the overall deployment of an InfluxDB high-availability system to achieve data consistency between two nodes, as described in this embodiment of the invention.

[0032] Figure 3 This is a schematic diagram of the core modules of an InfluxDB high-availability system as described in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram illustrating the processing procedure of the file storage component described in this embodiment of the invention;

[0034] Figure 5 This is a schematic diagram illustrating the processing procedure of the file synchronization component described in this embodiment of the invention;

[0035] Figure 6 This is a schematic diagram of the data writing process described in an embodiment of the present invention;

[0036] Figure 7 This is a schematic diagram of the data query process described in an embodiment of the present invention. Detailed Implementation

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0039] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0040] Terminology Explanation:

[0041] InfluxDB: An open-source time-series database for efficient storage, retrieval, and analysis of time-series data.

[0042] Synchronization Offset: Refers to the offset of the current synchronization during data synchronization.

[0043] Keepalive tool: Primarily used to detect server status, prevent single points of failure, and provide failover and load balancing functions.

[0044] SQL statements: A standard programming language specifically designed for managing relational databases, whose core functions include data querying, data manipulation, data definition, and data control.

[0045] TCP: A connection-oriented, reliable, byte-stream-based transport layer communication protocol, defined by IETF RFC 793.

[0046] In one or more embodiments, such as Figures 1-7 As shown, an InfluxDB high-availability system is provided, including:

[0047] The client is used to generate data query or write requests based on user needs and distribute them to the InfluxDB nodes;

[0048] InfluxDB nodes, including the middleware layer and InfluxDB, include:

[0049] The middle layer includes a service interface component, a file storage component, a file synchronization component, and a storage node component. For received data query requests, the middle layer forwards the data query request to the storage node component via the service interface component. The storage node component then encapsulates the current query request into a query request supported by the InfluxDB query interface, performs the InfluxDB data query, and returns the query results to the client. For received data write requests, the middle layer forwards the data to be written to the file storage component via the service interface component for local file writing. Simultaneously, the storage node component writes the data to be written to InfluxDB and returns the write status.

[0050] InfluxDB is used for data storage;

[0051] The InfluxDB nodes are configured in pairs, with each pair serving as both primary and backup nodes. When data is written, the data is synchronized between the primary and backup nodes through a file synchronization component.

[0052] InfluxDB high-availability systems utilize a local file automatic replication service to achieve data consistency. For example... Figure 2 As shown, after data is written to a local file on an InfluxDB node, it is automatically synchronized to other InfluxDB nodes to ensure data consistency in the local files. Each InfluxDB node deploys the influxdb-relay service, which maintains a local file. User read and write requests first request the influxdb-relay service, which then writes the data to the local file and simultaneously inserts it into the database. The influxdb-relay service includes components such as Httpserver, FileBackend, FileSync, and InfluxBackend, making it easy to manage and extend.

[0053] The automatic replication service synchronizes local files between InfluxDB nodes to ensure file data consistency, thereby achieving data consistency between two InfluxDB nodes.

[0054] In practical implementation, the data query or write request is generated based on user needs. The data query or write request uses a unified destination address for data query or writing. The Keepalive tool is used to monitor the operation status of the primary and backup nodes and automatically switch the virtual IP address to the InfluxDB node that is working properly.

[0055] Specifically, the solution described in this embodiment utilizes keepalive to achieve automatic VIP (virtual IP) switching. Users are unaware of the backend InfluxDB nodes, using a unified address for data querying and writing. Upon receiving a message, the service automatically routes it and sends the message directly to InfluxDB. When a database node goes offline, an online node is automatically selected, ensuring high availability of the service. The overall deployment diagram is shown below. Figure 1 As shown.

[0056] This embodiment uses VIP automatic switching and Keepalive technology to achieve automatic failover of virtual IPs, ensuring service continuity.

[0057] In specific implementation, the service interface component includes a data query interface, a Ping interface, a data write interface, and a status notification interface. The service interface component specifically performs the following processing steps: It uses the virtual IP routing service in the Keepalive tool to call the Ping interface to determine if the current InfluxDB node is running normally. If it is running normally, it selects the current node to perform data query or write; if it is running abnormally, it selects a backup node for data query or write. During data query, it calls the data query interface to forward the data query request to the storage node component and returns the query results obtained by the storage node component to the client. During data write, it uses the data write interface to forward the data to be written to the file storage component for local file writing and to the write channel of the storage node component.

[0058] In specific implementation, the file storage component performs the following process: when newly written data is received, it queries the offset address at the time of the latest synchronization and uses the offset address as the data writing position to write the local file.

[0059] Specifically, the file storage component, namely the FileBackend component, provides file writing and file management functions, such as... Figure 4 As shown, the specific processing steps include the following:

[0060] 1) File writing:

[0061] ① Listen to the initialization notification channel. Once the file synchronization initialization is complete, query the synchronization offset and write the synchronization completion offset to the file.

[0062] ② After initialization, upon receiving newly written data, query the latest synchronized offset and write it to the file. Simultaneously, write the data to the data write channel of the storage node component (i.e., the influxdbBackend component).

[0063] 2) File Management:

[0064] ① During initialization, check if there is a file. If there is no file, create the file. Set the file size to a fixed value. When writing data, check the file size. If the file size exceeds the set value, automatically create a new file.

[0065] ② Regularly scan all files in the path and periodically delete those files that have reached their maximum size and whose contents have been fully synchronized.

[0066] In specific implementation, the file synchronization component performs the following process: when a data write request selects a node, it automatically reads the local file of the current node and reads the file data according to the offset address of the latest synchronization completed in the local cache; it establishes a TCP connection between the current node and its corresponding backup node, and realizes the synchronization of local files between the two nodes based on TCP transmission.

[0067] Specifically, the file synchronization component, FleSync, provides file synchronization functionality. After service initialization, if an InfluxDB node is selected as the master node, it automatically reads the local files on the node, reads the file data according to the local cached synchronization offset, establishes a TCP connection with another InfluxDB node, and synchronizes the files. After service initialization, if an InfluxDB node is selected as the slave node, it creates a TCP client, establishes a connection with another InfluxDB node, and receives data when it detects data synchronization from the other node, writing the data to the local file.

[0068] Implement TCP server and TCP client, such as Figure 5 As shown.

[0069] 1) TCP server:

[0070] ① Upon receiving the master status signal, start the server and stop the client.

[0071] ② Receive synchronization data from the slave, write it to a file, and write it to the channel.

[0072] ③ Receive slave synchronization requests, read files, synchronize unsynchronized data, and update the synchronization offset. Send a synchronization initialization completion signal. (Initialization is considered complete if no connection is established within a timeout period).

[0073] ④ Synchronize the data received in the synchronization channel.

[0074] 2) TCP client:

[0075] ① Send a request to the master; after the connection is established, read the most recent data, get the offset, scan the file to synchronize any unsynchronized data, and request the master to synchronize the data.

[0076] ② Receive synchronous data and write it to the library channel.

[0077] In specific implementation, the storage node component performs the following processing steps: when a data query request is received, the request is encapsulated into a query request supported by the InfluxDB query interface, an InfluxDB query is performed, and the returned data is encapsulated and returned; when a data write request is received, the data write request is parsed, assembled into a corresponding data write SQL statement according to the InfluxDB SQL syntax, the data is written to InfluxDB based on the SQL statement, and the write status is returned.

[0078] In one or more embodiments, this embodiment provides a method for operating an InfluxDB high-availability system, which is based on the above-described InfluxDB high-availability system and includes:

[0079] Based on the destination address in the data query or write request sent by the client, select the InfluxDB node that can execute the current request normally;

[0080] In the selected InfluxDB node, for data query requests, the service interface component forwards the data query request to the storage node component. The storage node component then encapsulates the current query request into a query request supported by the InfluxDB query interface, performs the InfluxDB data query, and returns the query results to the client. For data write requests, the service interface component forwards the data to be written to the file storage component for local file writing. Simultaneously, the storage node component writes the data to be written to InfluxDB and returns the write status. When data is being written, the file synchronization component performs data synchronization between the primary and backup nodes.

[0081] In practice, a unified destination address is used for both data query and write requests. The Keepalive tool is used to monitor the operation of the primary and backup nodes and automatically switch the virtual IP address to the InfluxDB node that is working properly.

[0082] In practice, when a data write request selects a specific InfluxDB node, the local file of the current node is automatically read, and the file data is read in according to the offset address of the local cache when the latest synchronization was completed. A TCP connection is established between the current node and its corresponding backup node, and the synchronization of local files between the two nodes is achieved based on TCP transmission.

[0083] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A highly available InfluxDB system, characterized in that, include: The client is used to generate data query or write requests based on user needs and distribute them to the InfluxDB nodes; InfluxDB nodes, including the middleware layer and InfluxDB, include: The middle layer includes a service interface component, a file storage component, a file synchronization component, and a storage node component. For received data query requests, the middle layer forwards the data query request to the storage node component via the service interface component. The storage node component then encapsulates the current query request into a query request supported by the InfluxDB query interface, performs the InfluxDB data query, and returns the query results to the client. For received data write requests, the middle layer forwards the data to be written to the file storage component via the service interface component for local file writing. Simultaneously, the storage node component writes the data to be written to InfluxDB and returns the write status. The process generates data query or write requests based on user needs. In these requests, a unified destination address is used for data querying or writing. The Keepalive tool is used to monitor the operation of the primary and backup nodes and automatically switches the virtual IP address to a working InfluxDB node. The service interface component includes a data query interface, a Ping interface, a data write interface, and a status notification interface. Specifically, the service interface component performs the following processing steps: It uses the virtual IP routing function in the Keepalive tool to select a service call to the Ping interface to determine if the current InfluxDB node is running normally. If it is running normally, it selects the current InfluxDB node to perform data query or write; if it is running abnormally, it selects a backup node of the current InfluxDB node for data query or write. During data query, it calls the data query interface, forwards the data query request to the storage node component, and returns the query results obtained by the storage node component to the client. During data write, it uses the data write interface to forward the data to be written to the file storage component for local file writing and to the write channel of the storage node component. The file storage component specifically performs the following process: when newly written data is received, it queries the offset address at the time of the latest synchronization and uses the offset address as the data writing position to write the local file; The storage node component specifically performs the following processing steps: when a data query request is received, the request is encapsulated into a query request supported by the InfluxDB query interface, an InfluxDB query is performed, and the returned data is encapsulated and returned; when a data write request is received, the data write request is parsed, assembled into the corresponding SQL statement according to the InfluxDB SQL syntax, the data is written to InfluxDB based on the SQL statement, and the write status is returned. InfluxDB is used for data storage; The InfluxDB nodes are configured in pairs, with each pair of InfluxDB nodes serving as both primary and backup nodes. When data is written, the data is synchronized between the primary and backup nodes through a file synchronization component. The file synchronization component performs the following process: automatically reads the local file of the current node and reads the file data according to the offset address of the latest synchronization completed in the local cache; establishes a TCP connection between the current node and its corresponding backup node, and realizes the synchronization of local files between the two nodes based on TCP transmission.

2. The InfluxDB high-availability system as described in claim 1, characterized in that, After service initialization, if an InfluxDB node is selected as the master node, the file synchronization component will perform the following process: automatically read the files on the master node, read the file data according to the local cached synchronization offset, establish a TCP connection with another InfluxDB node, and synchronize the files. If an InfluxDB node is selected as a slave node, the file synchronization component performs the following process: it creates a TCP client, establishes a connection between the TCP client and another InfluxDB node, and when it detects data synchronization from the other party, it accepts the data and writes it to a local file.

3. A working method for an InfluxDB high-availability system, characterized in that, It is based on an InfluxDB high-availability system as described in any one of claims 1-2, comprising: Based on the destination address in the data query or write request sent by the client, select the InfluxDB node that can execute the current request normally; In the selected InfluxDB node, for data query requests, the service interface component forwards the data query request to the storage node component. The storage node component then encapsulates the current query request into a query request supported by the InfluxDB query interface, performs the InfluxDB data query, and returns the query results to the client. For data write requests, the service interface component forwards the data to be written to the file storage component for local file writing. Simultaneously, the storage node component writes the data to be written to InfluxDB and returns the write status. When data is being written, the file synchronization component performs data synchronization between the primary and backup nodes.

4. The working method of an InfluxDB high-availability system as described in claim 3, characterized in that, Both data query and write requests use a unified destination address for data querying or writing. The Keepalive tool monitors the operation status of the primary and backup nodes and automatically switches the virtual IP address to the InfluxDB node that is working properly.

5. The working method of an InfluxDB high-availability system as described in claim 3, characterized in that, When a data write request selects a specific InfluxDB node, it automatically reads the local file of the current node and reads the file data according to the offset address of the latest synchronization completed in the local cache; it establishes a TCP connection between the current node and its corresponding backup node, and realizes the synchronization of local files between the two nodes based on TCP transmission.

Images