A method and apparatus for replicating a time series database in parallel with sending requests

Abstract

The application relates to the technical field of time series database data replication, and particularly provides a time series database replication method and device for parallel sending of requests, wherein a client simultaneously establishes connections with master and backup database nodes, and simultaneously submits SQL statements to the master and backup databases during business processing, and then waits for the following two results to complete one business processing: (1) receiving the SQL statement execution result from the master database node; and (2) receiving the confirmation information that the SQL statement has been transmitted from the backup database node. Compared with the prior art, the application can improve the performance loss of the conventional master and backup data replication scheme in the synchronous mode and the data loss problem in the asynchronous replication mode.

Description

Technical Field

[0001] This invention relates to the field of time-series database data replication technology, specifically providing a time-series database replication method and apparatus for sending requests in parallel. Background Technology

[0002] Data replication technology is a crucial foundation for database migration and high availability solutions. For example, the common master-slave replication technology achieves the goal of creating dual copies of user-written data through continuous data replication between master and slave database nodes. This allows for rapid switchover to the database replica on the other node in the event of a failure in either the master or slave node, continuing to provide services. Common general-purpose databases, such as DB2 and MySQL, provide data replication functionality through kernel modules or external tools. Their data replication implementation methods can generally be divided into two main categories: synchronous replication and asynchronous replication. Their main characteristics and differences are as follows:

[0003] In synchronous replication mode, the master node must receive information from the slave node and confirm that the data has been replicated to the slave node before ending the business process and returning the client request. Synchronous replication mode can provide good guarantees of data integrity and consistency between data replicas, but at the same time, since each business process has to wait for the data replication between the master and slave database nodes to be completed, it has a certain impact on the performance of business processing.

[0004] In asynchronous replication mode, the master node immediately returns the database request to the client after executing the SQL statement. Simultaneously, a background asynchronous process replicates the business data to the standby node, so the data replication result does not affect subsequent client processing. Therefore, asynchronous replication mode has almost no impact on business processing performance. However, asynchronous mode requires additional processing logic to ensure the integrity and consistency of the data replicas between the master and standby databases, resulting in considerable complexity in implementation. Furthermore, in the event of a disaster, it is common for completed business processes on the master node to occur before the modified data has been replicated to the standby node, meaning that asynchronous replication mode carries the possibility of data loss. Compared to general-purpose databases, time-series databases typically have simpler business requests and often come with dedicated client drivers for connecting to the database.

[0005] Based on the above, there are issues such as the synchronous replication mode between the primary and standby databases leading to a decrease in business processing performance and affecting the continuous writing capability of time-series data; and the asynchronous replication mode between the primary and standby databases causing data loss, which may affect the continuous operation capability of the business after the primary-standby switchover. Summary of the Invention

[0006] This invention addresses the shortcomings of the prior art by providing a highly practical method for parallel sending requests in a time-series database replication process.

[0007] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable time-series database replication device for parallel sending requests.

[0008] The technical solution adopted by this invention to solve its technical problem is:

[0009] A time-series database replication method that sends requests in parallel involves the client simultaneously establishing connections with both the primary and backup database nodes, submitting SQL statements to both databases during business processing, and then completing one business process after receiving the following two results:

[0010] (1) Receive the SQL statement execution result from the master database node;

[0011] (2) Receive confirmation information from the backup database node that the SQL statement has been conveyed.

[0012] Furthermore, after receiving the SQL statement sent by the client, the master database node begins to execute the SQL statement, and returns the execution result to the client after the execution is completed;

[0013] At the same time, the primary database node also needs to notify the backup database node that the SQL statement has been executed and confirm that the backup database node accepts the message.

[0014] Furthermore, after receiving the SQL statement sent by the client, the backup database node first writes the SQL statement into the cache area, and then returns a message to the client to confirm that the SQL statement has been received and saved.

[0015] Furthermore, another process in the backup database node is used to receive the SQL execution completion message sent by the master database process. After receiving the execution completion message from the master database, it finds the corresponding SQL in the cache and then executes the SQL statement on the backup data node, thus completing the data replication.

[0016] Furthermore, the client establishes a connection with the primary database node and queries whether a backup node exists and the connection method of the backup node. If the primary node returns the connection method of the backup node, then a connection with the backup database node is established.

[0017] Upon receiving the SQL statement from the user's business request, a UUID is first generated as a globally unique identifier for the SQL statement. The SQL statement and its identifier UUID are then sent to both the primary and backup database nodes. The system waits for the execution result returned by the primary node and for confirmation from the backup node that the SQL statement has been cached.

[0018] Furthermore, when the primary database node receives a new connection request from the client, it proactively returns the backup node connection information, executes the SQL statement upon receiving it, and returns the execution result to the client.

[0019] If the SQL statement is executed successfully, a message is sent to the backup node to notify it that the SQL statement with the UUID number has been executed.

[0020] Furthermore, when the backup database node receives an SQL statement sent by the client, it saves the SQL statement to the cache and returns a confirmation message to the client that the SQL statement result has been cached.

[0021] When a notification message is received from the master database node informing that the SQL statement with the number UUID has been executed, the SQL statement is retrieved from the cache and executed. If the execution is successful, the data replication is complete and the SQL statement can be deleted from the cache.

[0022] If the execution fails, the master data node and all connected clients will be notified that a data replication error has occurred.

[0023] A time-series database replication apparatus for parallel transmission of requests includes: at least one memory and at least one processor;

[0024] The at least one memory is used to store a machine-readable program;

[0025] The at least one processor is configured to invoke the machine-readable program to execute a time-series database replication method for sending requests in parallel.

[0026] Compared with the prior art, the time-series database replication method and apparatus for parallel request transmission of the present invention has the following outstanding advantages:

[0027] This invention establishes connections with both primary and backup database nodes simultaneously using a dedicated driver for time-series data clients, and sends business request SQL statements to both nodes at the same time.

[0028] Furthermore, by transmitting SQL statement execution completion messages between the primary and standby nodes, the goal of jointly executing SQL statements on both nodes is achieved, thus completing data replication. Since the SQL requests are sent simultaneously to both the primary and standby databases by the client driver and are only cached in the standby database, this approach improves upon the performance loss in synchronous mode and the data loss issues in asynchronous replication, as is common in traditional primary-standby data replication schemes. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Appendix Figure 1 This is a flowchart illustrating a time-series database replication method that sends requests in parallel. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] The following is a preferred embodiment:

[0033] like Figure 1 As shown in this embodiment, a time-series database replication method for sending requests in parallel involves the client simultaneously establishing connections with the primary and backup database nodes and submitting SQL statements to both the primary and backup databases during business processing. After receiving the following two results, a business process is completed:

[0034] (1) Receive the SQL statement execution result from the master database node;

[0035] (2) Receive confirmation information from the backup database node that the SQL statement has been conveyed.

[0036] The primary database node executes the SQL statement sent by the client upon receipt and returns the execution result to the client upon completion. Simultaneously, the primary database node must notify the backup database node that the SQL statement has been executed and confirm that the backup database node accepts the message.

[0037] After receiving an SQL statement from the client, the standby database node first writes the SQL statement into a cache. After caching, it returns a message to the client confirming receipt and saving of the SQL statement. Simultaneously, another process within the standby database node receives SQL execution completion messages from the primary database process. Upon receiving this message, the standby node retrieves the corresponding SQL statement from the cache and executes it, thus completing the data replication.

[0038] From the client's perspective;

[0039] The client establishes a connection to the primary database node and queries whether a backup node exists and the connection method for the backup node. If the primary node returns the connection method for the backup node, then a connection to the backup database node is established.

[0040] Upon receiving the SQL statement from the user's business request, a UUID is first generated as a globally unique identifier for the SQL statement. The SQL statement and its identifier UUID are then sent to both the primary and backup database nodes. The system waits for the execution result returned by the primary node and for confirmation from the backup node that the SQL statement has been cached.

[0041] For the primary database node;

[0042] When the primary database node receives a new connection request from the client, it proactively returns the backup node connection information. Upon receiving the SQL statement, it executes the SQL statement and returns the execution result to the client.

[0043] If the SQL statement is executed successfully, a message is sent to the backup node to notify it that the SQL statement with the UUID number has been executed.

[0044] For backup database nodes;

[0045] When the backup database node receives an SQL statement from the client, it saves the SQL statement to the cache and returns a confirmation message to the client that the SQL statement result has been cached.

[0046] When a notification message is received from the master database node informing that the SQL statement with the number UUID has been executed, the SQL statement is retrieved from the cache and executed. If the execution is successful, the data replication is complete and the SQL statement can be deleted from the cache.

[0047] If the execution fails, the master data node and all connected clients will be notified that a data replication error has occurred.

[0048] Based on the above method, a time-series database replication apparatus for parallel sending requests in this embodiment includes: at least one memory and at least one processor;

[0049] The at least one memory is used to store a machine-readable program;

[0050] The at least one processor is configured to invoke the machine-readable program to execute a time-series database replication method for sending requests in parallel.

[0051] The specific embodiments described above are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the specific embodiments described above. Any appropriate changes or substitutions made by a person skilled in the art that conform to the claims of the present invention regarding a time-series database replication method and apparatus for parallel transmission requests should fall within the patent protection scope of the present invention.

[0052] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A time-series database replication method for parallel request sending, characterized in that, The client simultaneously establishes connections with the primary and standby database nodes, and submits the SQL statement to both the primary and standby databases during business processing. It then waits to receive the following two results before completing a business process: (1) Receive the SQL statement execution result from the main database node; (2) Receive confirmation information from the backup database node that the SQL statement has been transmitted; The master database node starts executing the SQL statement after receiving it from the client, and returns the execution result to the client after the execution is completed. At the same time, the primary database node also needs to notify the backup database node that the SQL statement has been executed and confirm that the backup database node accepts the message. After receiving the SQL statement sent by the client, the backup database node first writes the SQL statement into the cache area, and then returns a message to the client to confirm that the SQL statement has been received and saved. Meanwhile, another process in the backup database node is used to receive the SQL execution completion message sent by the master database process. After receiving the execution completion message from the master database, it finds the corresponding SQL in the cache and then executes the SQL statement on the backup data node, thus completing the data replication. The client establishes a connection with the primary database node and queries whether a backup node exists and the connection method of the backup node. If the primary node returns the connection method of the backup node, then a connection with the backup database node is established. After receiving the SQL statement in the user's business request, a UUID is first generated as a globally unique ID to identify the SQL statement in the future. The SQL statement and the UUID that identifies it are sent to the primary and backup database nodes at the same time. Waiting to receive the execution result returned by the primary node and waiting to receive the confirmation message from the backup node that the SQL statement has been cached. When the primary database node receives a new connection request from the client, it proactively returns the backup node connection information, executes the SQL statement upon receiving it, and returns the execution result to the client. If the SQL statement is executed successfully, a message is sent to the backup node to notify it that the SQL statement with the UUID number has been executed. When the backup database node receives an SQL statement sent by the client, it saves the SQL statement to the cache and returns a confirmation message to the client that the SQL statement result has been cached. When a notification message is received from the master database node informing that the SQL statement with the number UUID has been executed, the SQL statement is retrieved from the cache and executed. If the execution is successful, the data replication is complete and the SQL statement can be deleted from the cache. If the execution fails, the master data node and all connected clients will be notified that a data replication error has occurred.

2. A time-series database replication apparatus for parallel request transmission, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to execute the method of claim 1.

Images