A secure stopping method and system based on a distributed database

Abstract

The application relates to the technical field of distributed databases, in particular to a safe stopping method and system based on a distributed database, which comprises the following steps: setting a monitoring device, capturing a safe closing instruction in real time, and sending a safe stopping task request to a database master node; when the request receiving is successful, entering a safe stopping mode, first executing a command of stopping all timing tasks; then executing a safe stopping task command on each subnode; and periodically checking whether each subnode meets a safe stopping condition, if yes, ending a database process and releasing resources; if not, returning to continue executing the safe stopping task until all the subnodes meet the safe stopping condition; for a transaction executed beyond a time limit, a time limit early warning mechanism is used to reasonably stop transaction execution; the application realizes the function of safe stopping of the distributed database, guarantees the safety and integrity of data, and improves the practicability of the distributed database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed database, and particularly relates to a safe stopping method and system based on a distributed database. BACKGROUND

[0002] With the development of business, the amount of data generated will become very large, when a single machine is difficult to support, an expansion bottleneck will be faced, so the data needs to be split and dispersed on multiple database nodes. Unlike centralized databases, in a distributed database cluster, to shut down all nodes of the entire cluster, the shutdown command needs to be issued to each node, and each node needs to be shut down. In this process, there may be some nodes with incomplete transactions, or new transactions are started on a certain node. Directly executing kill-9 forced shutdown will cause the transactions executed by a certain node to be abnormal, resulting in dirty data, and even causing the entire service to be abnormal and unable to start.

[0003] A device, system and method for data recovery in disaster recovery are disclosed in Chinese patent application CN113010348A. The device for data recovery in disaster recovery includes a first backup database storing first backup data from a production database, and a second backup database storing second backup data from the production database. In the case where the production database stops working, the first backup data is used to continue working, and the second backup data is used to supplement the first backup data. The method for data recovery in disaster recovery includes storing first backup data from a production database in a first backup database, storing second backup data from the production database in a second backup database, continuing to work based on the first backup data in the case where the production database stops working, and supplementing the first backup data using the second backup data.

[0004] The above prior art has a native distributed database system. When shutting down the entire cluster, the command is directly issued to each database node, and the program is directly closed. When the service is executing a transaction, it may cause some nodes to succeed and some nodes to fail. At this time, without a recovery mechanism, the incomplete transactions will be directly discarded, causing data anomalies, which may cause serious consequences. In order to solve these problems, the present application provides a safe stopping method and system based on a distributed database. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a safe stopping method based on a distributed database, which not only realizes the fast and safe stopping of the distributed database, but also guarantees the consistency of the data in the distributed database.

[0006] To achieve the above object, the present application provides the following technical solutions:

[0007] A security stopping method based on a distributed database, comprising the following steps:

[0008] S1: A monitoring device is set, when an exception occurs in the database or an external closing instruction is received, the monitoring device captures a security closing instruction in real time, and performs verification and authorization operations on the instruction, and sends a security stopping task request to the master node of the distributed database when the authorization is passed;

[0009] S2: When the master node of the distributed database successfully receives and parses the security stopping task request, the database enters a security stopping mode; otherwise, the security stopping task request is failed, and the task request is ended;

[0010] S3: After entering the security stopping mode, according to the security stopping task request sent by the master node to each sub-node, first, the command of stopping all timing tasks is executed, and all timing tasks are stopped; then, the security stopping task is executed, and whether each sub-node meets the security stopping condition is detected, if yes, the database process is ended, and if not, the security stopping task is returned to continue to be executed, and a timeout warning mechanism is set, transactions executed beyond the timeout are stopped, and the transactions are not invalidated;

[0011] S4: When all the connection transactions of the sub-nodes are successfully completed, the sub-nodes send the execution results to the master node; the master node executes an instruction to end the database process according to the feedback, and notifies each sub-node to release resources, so that the entire distributed database system enters a security stopping state.

[0012] Further, the monitoring device receives the closing instruction through subscribing to the closing signal of the database system or through a special API interface.

[0013] Further, the timing task includes heartbeat detection and timing backup.

[0014] Further, the security stopping task in S3 is as follows:

[0015] S4.1: When a user issues an SQL request to the distributed database node through a client to establish a new front-end connection, the new front-end connection is refused through an authentication mechanism, and the database has stopped is returned, and new transactions are prevented from being generated;

[0016] The authentication mechanism is realized through user identity verification, when the security stopping mode is entered, the establishment of the new front-end connection is realized by refusing the identity verification request of the new user, and the established front-end connection user can pass the identity verification request;

[0017] S4.2: When a user sends a new SQL request to the database node through an existing front-end connection, a filtering mechanism is used to determine whether the new SQL request is in a transaction.

[0018] The filtering mechanism is executed through the parameter isAutocommit. When the isAutocommit parameter is True, the new SQL request is in a transaction and the request is allowed to be executed until the transaction is completed. When the isAutocommit parameter is false, the new SQL request is not in a transaction and the request is rejected, and a graceful shutdown notification is provided.

[0019] Furthermore, the specific steps in S3 to check whether the safe stop conditions are met are as follows:

[0020] S5.1: After completing a safe stop task, each child node performs an asynchronous scan operation at regular intervals and calculates the total number of front-end connection transactions still existing in each child node;

[0021] S5.2: If the total number of front-end connection transactions is 0, then proceed to the back-end transaction statistics stage to count the total number of all back-end connection transactions; otherwise, return to wait; if the total number of back-end connection transactions is also 0, then the safe stop condition is met; otherwise, return to wait.

[0022] Furthermore, the timeout warning mechanism is as follows:

[0023] S6.1: When any child node exceeds the timeout threshold for executing a safe stop task, and there are still unfinished transactions under any child node, a rollback operation is performed to stop the transaction, and the transaction is marked as an incomplete transaction, and the mark is recorded in the rollback segment; when the database is restarted, the transaction status in the rollback segment is checked through SQL commands, and the incomplete transaction is re-executed.

[0024] S6.2: When the transaction is a local transaction, the rollback operation is performed using the rollback mechanism provided by the database itself under the child node, thereby realizing the rollback of the transaction;

[0025] S6.3: When the transaction is a distributed transaction, the distributed transaction manager Seata coordinates the child nodes and sends a rollback instruction to the resource manager of the child node whose transaction execution has not been completed. The resource manager executes the rollback operation by calling the database XA interface under the child node whose transaction execution has not been completed, thus realizing the rollback of the transaction.

[0026] Furthermore, in S5.1, the time interval is 5 seconds.

[0027] Furthermore, the timeout warning mechanism is set to a timeout threshold of 30 seconds.

[0028] A safe shutdown system based on a distributed database includes a monitoring module, a safe shutdown module, a safe shutdown detection module, a timeout warning module, and a resource release module;

[0029] The monitoring module is used to capture safe shutdown commands in real time and send safe stop task requests to lower-level child nodes of the database;

[0030] The safe stop module is used to perform safe stop tasks on each child node of the database;

[0031] The safe stop detection module is used to detect whether each child node of the database meets the safe stop conditions;

[0032] The timeout warning module is used to detect and stop transactions that take longer than the timeout threshold in the safe stop mode.

[0033] The resource release module is used to release database resources and complete a safe shutdown.

[0034] Furthermore, the safety stop detection module includes a front-end connection detection subunit and a back-end connection detection subunit;

[0035] The front-end connection detection subunit is used to count whether all front-end connection transactions under each child node have been completed and meet the safe stopping conditions;

[0036] The backend connection detection subunit is used to count whether all backend connection transactions under each child node have been completed and meet the safe stopping conditions.

[0037] A storage medium storing instructions that, when read by a computer, cause the computer to execute a secure shutdown method based on a distributed database as described in any of the preceding claims.

[0038] An electronic device includes a processor and the aforementioned storage medium, the processor executing instructions stored in the storage medium.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] 1. This invention modifies the original forced stop command into a safe stop method, achieving the completion of triggered transactions and data integrity while maintaining the original functionality, request method, and minimal performance loss;

[0041] 2. This invention constructs a safe stop mode in the database, ensuring that existing transactional connections do not become invalid, while non-transactional connections return a graceful service stop prompt, thus guaranteeing data consistency after the database stops.

[0042] 3. By introducing a timeout warning mode into the safe stop mode, this invention ensures both the rapid stopping of long-running transactions and the preservation of the transaction's validity, thereby improving the efficiency and practicality of the method. Attached Figure Description

[0043] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0044] Figure 1 This is a schematic diagram of a safe shutdown method based on a distributed database according to Embodiment 1 of the present invention;

[0045] Figure 2 This is a logic structure diagram of the detection safety stop condition in Embodiment 1 of the present invention;

[0046] Figure 3 This is a structural diagram illustrating the method verification based on the Tianyi Cloud distributed database udal in Embodiment 2 of the present invention.

[0047] Figure 4 This is a block diagram of a safe shutdown system based on a distributed database, according to Embodiment 3 of the present invention. Detailed Implementation

[0048] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0049] Example 1:

[0050] Currently, there are many ways to stop a distributed database, including two-phase shutdown, phased shutdown, and sharded shutdown. However, most of these methods have the problems mentioned in the background above. For example, the sharded shutdown method may result in some shards being shut down while others are still running, causing data inconsistency. Forcibly shutting down a database that is still running may also cause data loss.

[0051] To address the aforementioned problems, this invention provides an embodiment: a secure shutdown method based on a distributed database. Please refer to [link / reference]. Figure 1 The specific steps include:

[0052] S1: Set up a monitoring device. When the database encounters an anomaly or receives an external shutdown command, the monitoring device will capture the safe shutdown command in real time, perform verification and authorization operations on the command, and send a safe stop task request to the distributed database master node after authorization is successful.

[0053] S2: When the distributed database master node successfully receives and parses the safe stop task request, the database enters the safe stop mode; otherwise, if the task request fails to be received, the task request is terminated.

[0054] S3: After entering the safe stop mode, according to the safe stop task request sent by the master node to each child node, first execute the command to stop all scheduled tasks and stop all scheduled tasks; then, execute the safe stop task and check whether each child node meets the safe stop condition. If it meets the condition, the database process will end; if it does not meet the condition, it will return to continue executing the safe stop task and set a timeout warning mechanism to stop transactions that have timed out, while ensuring that the transaction does not become invalid.

[0055] S4: When all connection transactions of child nodes have been successfully completed, the child nodes send the execution results to the master node; the master node executes the SQL command to terminate the database process based on the feedback, and notifies each child node to release resources, so that the entire distributed database system enters a safe shutdown state.

[0056] Specifically, the monitoring equipment receives shutdown commands either by subscribing to shutdown signals from the database system or through a dedicated API interface.

[0057] Specifically, the S3 safe stop task is as follows:

[0058] S4.1: When a user sends an SQL request to a distributed database node through a client to establish a new front-end connection, the authentication mechanism rejects the new front-end connection and returns a message indicating that the database has stopped, preventing new transactions from occurring.

[0059] The above authentication mechanism is implemented through user authentication. When entering the secure stop mode, a new front-end connection is established by rejecting the authentication request of a new user. Users with an established front-end connection can then make authentication requests.

[0060] S4.2: When a user sends a new SQL request to the database node through an existing front-end connection, a filtering mechanism is used to determine whether the new SQL request is in a transaction.

[0061] The filtering mechanism is executed through the parameter isAutocommit. When the isAutocommit parameter is True, the new SQL request is in a transaction and the request is allowed to be executed until the transaction is completed. When the isAutocommit parameter is false, the new SQL request is not in a transaction and the request is rejected, and a graceful shutdown notification is provided.

[0062] Specifically, please refer to Figure 2 The specific steps for checking whether the safe stop conditions are met in S3 are as follows:

[0063] S5.1: After completing a safe stop task, each child node performs an asynchronous scan operation at intervals and calculates the total number of front-end connection transactions still existing in each child node;

[0064] S5.2: If the total number of front-end connection transactions is 0, proceed to the back-end transaction statistics stage to calculate the total number of back-end connection transactions still existing in each child node; otherwise, return to wait and repeat S5.1; if the total number of back-end transactions is also 0, the safe stop condition is met; otherwise, return to wait.

[0065] Specifically, the timeout warning mechanism is as follows:

[0066] S6.1: When each child node exceeds the timeout threshold for executing the safe stop task, and there are still unfinished transactions under any child node, a rollback operation is performed to stop the transaction, and the transaction is marked as an incomplete transaction, and the mark is recorded in the rollback segment; when the database is restarted, the transaction status in the rollback segment is checked through SQL commands, and the incomplete transaction is re-executed.

[0067] S6.2: When the transaction is a local transaction, the rollback operation is performed using the rollback mechanism provided by the database itself under the child node, thereby realizing the rollback of the transaction.

[0068] S6.3: When the transaction is a distributed transaction, the distributed transaction manager Seata coordinates each child node and sends a rollback instruction to the resource manager of the child node whose transaction execution has not been completed. The resource manager executes the rollback operation by calling the database XA interface under the child node whose transaction execution has not been completed, thereby realizing the rollback of the transaction.

[0069] Specifically, in S5.1, a period of time is 5 seconds.

[0070] Specifically, the timeout warning mechanism is set to a timeout threshold of 30 seconds.

[0071] Example 2:

[0072] To verify the secure shutdown method based on a distributed database provided in Example 1, this invention deployed a specific embodiment on the Tianyi Cloud distributed database udal. Please refer to [link to example]. Figure 3 Assuming that the Tianyi Cloud distributed database udal includes two nodes: dbproxy1 and dbproxy2, each node is associated with two Tianyi Cloud database rds instances, and that network connectivity and normal operation between nodes of each instance are guaranteed; the specific steps of this embodiment are as follows:

[0073] S2.1: Establish a connection on dbproxy1 and execute a normal select statement to simulate a non-transactional query.

[0074] S2.2: Establish a connection on dbproxy2 and execute the command udal dt start to start a distributed transaction, simulating a transaction query state.

[0075] S2.3: Send the secure stop command kill-15 through the Tianyi Cloud distributed database client, then capture the SQL command in real time through the monitoring device, perform verification and authorization operations on the command, send the authorized command to the database master node, and determine whether the command was successfully received. If it is successful, enter the secure stop mode, stop all scheduled tasks, and execute the secure stop task; if it fails, end the secure stop request.

[0076] S2.4: After successfully entering safe mode, first send a request to establish a new front-end connection to either dbproxy1 or dbproxy2 through the client port. The request will be rejected and the system will return "Database safe stop" and refuse the connection.

[0077] S2.5: When a user inputs a non-transactional SQL query command into the client and performs a non-transactional query operation on any database rds1 or rds2 under node dbproxy1 through the connection established in S1, it will return to "Safely Stopping" and disconnect from node dbproxy1.

[0078] S2.6: The user inputs a transaction query SQL command to the client. Through the connection established in S2, a transaction query operation is performed on any database rds3 or rds4 under node dbproxy2. The query will return a success message and the queried data will be fed back to the client. When all transactions in databases rds3 and rds4 under this node are completed, the system returns to safe stop, releases resources, and disconnects from node dbproxy2. If the transaction execution time under dbproxy2 exceeds 30 seconds, the timeout warning mechanism is used to roll back the incomplete transaction and mark it as an incomplete transaction, and the mark is recorded in the rollback segment. When the database is restarted, the transaction status in the rollback segment is checked through SQL commands, and the incomplete transaction is re-executed.

[0079] Specifically, the steps to establish a connection on dbproxy1 and simulate a non-transactional query are as follows:

[0080] S1.1: In the China Telecom Cloud Distributed Database Client, enter the domain name, database port, username and password of dbproxy1 to establish a connection with the dbproxy1 node;

[0081] S1.2: After a successful connection, execute a regular SELECT statement without transaction encapsulation to query the data rds in any database under dbproxy1, simulating a non-transactional query; for example, SELECT * FROM rds1 means querying the data in the first associated database under dbproxy1.

[0082] Specifically, the steps for establishing a connection on dbproxy2 and simulating a transaction query are as follows:

[0083] S2.1: In the China Telecom Cloud Distributed Database Client, enter the domain name, database port, username and password of dbproxy2 to establish a connection with the dbproxy2 node;

[0084] S2.2: Connection established successfully. Execute the command udal dt start to start the distributed transaction processing mechanism;

[0085] S2.3: After enabling distributed transactions, execute SQL statements including the transaction encapsulation statements BEGIN and COMMIT to query any associated database rds on the dbproxy2 node, simulating a transaction query; for example:

[0086] BEGIN TRANSACTION: Begins a transaction.

[0087] SELECT * FROM table_name WHERE condition, execute the query operation;

[0088] COMMIT commits the transaction, saving all operations to the database.

[0089] Specifically, the execution process of using a timeout warning mechanism to roll back incomplete transactions and ensure that the transactions do not become invalid is as follows:

[0090] When the transaction is executed only on either database rds3 or database rds4, the rollback operation can be performed using the built-in rollback mechanism of either database rds3 or database rds4 to achieve the transaction rollback.

[0091] When the transaction is a distributed transaction and is executed on both database rds3 and database rds4, the distributed transaction manager Seata sends a rollback command to the resource manager of the dbproxy2 node. The resource manager executes the rollback operation by calling the XA interface of database rds3 or database rds4 under dbproxy2, or calling both database rds3 and database rds4 simultaneously, thus realizing the rollback of the transaction.

[0092] Example 3:

[0093] Please see Figure 4 A safe stop system based on a distributed database includes a monitoring module, a safe stop module, a safe stop detection module, a timeout warning module, and a resource release module;

[0094] The monitoring module is used to capture safe shutdown commands in real time and send safe stop task requests to lower-level child nodes of the database;

[0095] The safe stop module is used to perform safe stop tasks on each child node of the database;

[0096] The safe stop detection module is used to detect whether each child node of the database meets the safe stop conditions;

[0097] The timeout warning module is used to detect and stop transactions that take longer than the timeout threshold in the safe stop mode.

[0098] The resource release module is used to release database resources and complete a safe shutdown.

[0099] Furthermore, the safety stop detection module includes a front-end connection detection subunit and a back-end connection detection subunit;

[0100] The front-end connection detection subunit is used to count whether all front-end connection transactions under each child node have been completed and meet the safe stopping conditions;

[0101] The backend connection detection subunit is used to count whether all backend connection transactions under each child node have been completed and meet the safe stopping conditions.

[0102] Example 4:

[0103] A storage medium storing instructions that, when read by a computer, cause the computer to execute a secure shutdown method based on a distributed database.

[0104] Example 5

[0105] An electronic device includes a processor and a storage medium, wherein the processor executes an unsupervised real-time low-light image enhancement method based on a self-calibration mechanism in the storage medium.

[0106] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A safe stopping method based on a distributed database, characterized in that, Includes the following steps: S1: Set up a monitoring device. When the database encounters an anomaly or receives an external shutdown command, the monitoring device will capture the safe shutdown command in real time, perform verification and authorization operations on the command, and send a safe stop task request to the distributed database master node after authorization is successful. S2: When the distributed database master node successfully receives and parses the safe stop task request, the database enters the safe stop mode; otherwise, if receiving the safe stop task request fails, the task request is terminated. S3: After entering the safe stop mode, according to the safe stop task request sent by the master node to each child node, first execute the command to stop all scheduled tasks and stop all scheduled tasks; then, execute the safe stop task and check whether each child node meets the safe stop condition. If it meets the condition, the database process will end; if it does not meet the condition, it will return to continue executing the safe stop task and set a timeout warning mechanism to stop transactions that have timed out, while ensuring that the transaction does not become invalid. The specific details of the S3 safe stop task are as follows: S4.1: When a user sends an SQL request to a distributed database node through a client to establish a new front-end connection, the authentication mechanism rejects the new front-end connection and returns a message indicating that the database has stopped, preventing new transactions from occurring. The authentication mechanism is implemented through user authentication. When entering the secure stop mode, a new front-end connection is established by rejecting the authentication request of a new user. Users with an established front-end connection can then make authentication requests. S4.2: When a user sends a new SQL request to the database node through an existing front-end connection, a filtering mechanism is used to determine whether the new SQL request is in a transaction. The filtering mechanism is executed through the parameter isAutocommit. When the isAutocommit parameter is True, the new SQL request is in a transaction and the request is allowed to be executed until the transaction is completed. When the isAutocommit parameter is false, the new SQL request is not in a transaction and the request is rejected, and a graceful close notification is returned. The specific steps in S3 to detect whether the safe stopping condition is met are as follows: S5.1: After completing a safe stop task, each child node performs an asynchronous scan operation at intervals and calculates the total number of front-end connection transactions still existing in each child node; S5.2: If the total number of front-end connection transactions is 0, then proceed to the back-end transaction statistics stage to count the total number of all back-end connection transactions; otherwise, return to wait. If the total number of back-end connection transactions is also 0, then the safe stop condition is met; otherwise, return to wait. S4: When all connection transactions of child nodes have been successfully completed, the child nodes send the execution results to the master node; based on the feedback, the master node executes the instruction to terminate the database process and notifies each child node to release resources, so that the entire distributed database system enters a safe shutdown state.

2. The secure shutdown method based on a distributed database according to claim 1, characterized in that, The monitoring equipment receives shutdown commands either by subscribing to shutdown signals from the database system or through a dedicated API interface.

3. The secure shutdown method based on a distributed database according to claim 2, characterized in that, The scheduled tasks include heartbeat detection and scheduled backup.

4. The secure shutdown method based on a distributed database according to claim 3, characterized in that, The specific steps of the timeout warning mechanism are as follows: S6.1: When each of the child nodes exceeds the timeout threshold in executing the safe stop task, and there are still unfinished transactions under any child node, a rollback operation is performed to stop the transaction, and the transaction is marked as an incomplete transaction, and the mark is recorded in the rollback segment; When the database is restarted, the transaction status in the rollback segment is checked using SQL commands, and the incomplete transaction is re-executed. S6.2: When the transaction is a local transaction, the rollback operation is performed using the rollback mechanism provided by the database itself under the child node, thereby realizing the rollback of the transaction; S6.3: When the transaction is a distributed transaction, the distributed transaction manager Seata coordinates each child node and sends a rollback instruction to the resource manager of the child node whose distributed transaction has not been completed. The resource manager executes the rollback operation by calling the database XA interface under the child node, thereby realizing the rollback of the transaction.

5. A secure shutdown method based on a distributed database according to claim 4, characterized in that, In S5.1, the time period is 5 seconds.

6. A secure shutdown method based on a distributed database according to claim 5, characterized in that, The timeout warning mechanism is set to a timeout threshold of 30 seconds.

7. A secure shutdown system based on a distributed database, implemented based on a secure shutdown method based on a distributed database as described in any one of claims 1-6, characterized in that, The system includes a monitoring module, a safety stop module, a safety stop detection module, a timeout warning module, and a resource release module; The monitoring module is used to capture the safe shutdown command in real time and send a safe stop task request to the lower-level child nodes of the database; The safe stop module is used to perform safe stop tasks on each sub-node of the database; The safe stop detection module is used to detect whether each sub-node of the database meets the safe stop conditions; The timeout warning module is used to capture and roll back transactions that have exceeded the timeout threshold in the safe stop mode. The resource release module is used to perform resource release operations on each sub-node when the database reaches the safe stopping condition.

8. A secure shutdown system based on a distributed database according to claim 7, characterized in that, The safety stop detection module includes a front-end connection detection subunit and a back-end connection detection subunit; The front-end connection detection subunit is used to count whether all front-end connection transactions under each sub-node have been completed and meet the safe stopping conditions. The backend connection detection subunit is used to count whether all backend connection transactions under each sub-node have been completed and meet the safe stopping conditions.

9. A storage medium, characterized in that, The storage medium stores instructions that, when read by a computer, cause the computer to execute a secure stop method based on a distributed database as described in any one of claims 1-6.

10. An electronic device, characterized in that, The device includes a processor and the storage medium of claim 9, wherein the processor executes a secure shutdown method for a distributed database as described in any one of claims 1-6.

Images