Consistency checking method and device of database, electronic equipment and storage medium

Abstract

The disclosure provides a database consistency verification method and device, electronic equipment and computer readable storage medium, and relates to the field of databases. The database consistency verification method comprises: in response to a verification instruction, performing a verification operation on a first backup library and a second backup library based on a checksum algorithm, wherein a reference code is generated based on the verification operation for the first backup library, and a verification code is generated based on the verification operation for the second backup library; determining the consistency between the first backup library and the second backup library based on the comparison result of the reference code and the verification code; and determining the consistency between the master library and the backup library based on the consistency between the first backup library and the second backup library. Through the technical scheme of the disclosure, the consistency verification can be performed while ensuring the normal operation of the master library, which is conducive to reducing the probability of intrusion into the database, thereby ensuring the high availability and strong consistency of the database.

Description

Technical Field

[0001] This disclosure relates to the field of database technology, and in particular to a database consistency verification method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, data synchronization between MySQL database instances on the platform uses logical replication. However, replication based on logical SQL statements or logical rows cannot guarantee complete data consistency between the master and slave databases. For example, inconsistencies may occur after human error or server failure and recovery. Therefore, it is necessary to regularly check the master and slave data to promptly identify and fix any inconsistencies.

[0003] In related technologies, pt-table-checksum is used as a tool to verify master-slave consistency. pt-table-checksum runs on the master database, uses the CRC32 algorithm to perform database consistency verification, relies on the statement interface to obtain the result of whether the master and slave are consistent, and needs to write the verification configuration information and verification results into the production database. This solution currently has the following drawbacks:

[0004] Calling the statement interface and writing data are both intrusive to the database. In addition, if there is an error in the replication between the primary and standby databases, the verification tools will not be able to function properly.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a database consistency verification method, a database consistency verification device, an electronic device, and a computer-readable storage medium, which at least to some extent overcomes the problem of intrusiveness to databases in related technologies.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to one aspect of this disclosure, a database consistency verification method is provided, the database including a primary database and a backup database of the primary database, the backup database including a first backup database and a second backup database, comprising: responding to a verification command, performing verification operations on the first backup database and the second backup database respectively based on a checksum algorithm, wherein a reference code is generated for the first backup database based on the verification operation, and a checksum is generated for the second backup database based on the verification operation; determining the consistency between the first backup database and the second backup database based on the comparison result of the reference code and the checksum; and determining the consistency between the primary database and the backup database based on the consistency between the first backup database and the second backup database.

[0009] In one embodiment, the verification operation includes an initial verification operation. The verification operation based on the checksum algorithm is performed on the first backup database and the second backup database respectively, specifically including: when performing the initial verification operation on the first backup database, traversing all first data tables in the first backup database and calculating the checksum of all first data tables in the first backup database as an initial reference code; when performing the initial verification operation on the second backup database, traversing all second data tables in the second backup database and calculating the checksum of all second data tables in the second backup database as an initial checksum.

[0010] In one embodiment, the verification operation further includes a re-verification operation. Specifically, performing the verification operation on the first backup database and the second backup database based on the checksum algorithm further includes: when an inconsistency is detected between the initial reference code and the initial checksum, querying the inconsistent bit value fields in the initial reference code and the initial checksum based on the comparison result; determining all reference data tables in the first data tables that need to be re-verified, and all verification data tables in the second data tables that need to be re-verified based on the inconsistent bit value fields; and performing the re-verification operation on the reference data tables and the verification data tables.

[0011] In one embodiment, performing the re-verification operation on the reference data table and the verification data table specifically includes: updating the verification sum of the reference data table based on a preset update frequency, as an update reference code; and updating the verification sum of the verification data table based on the update frequency, as an update verification code. Determining the consistency between the first backup database and the second backup database based on the comparison result of the reference code and the verification code specifically includes: comparing the update reference code and the update verification code after each update; when the update count is less than a first threshold and the update reference code and the update verification code are detected to be consistent, determining that the first backup database and the second backup database are consistent; when the update count is greater than or equal to the first threshold and less than or equal to a second threshold, and the update reference code and the update verification code are still inconsistent, determining that the first backup database and the second backup database are inconsistent, and the second threshold is greater than or equal to the first threshold.

[0012] In one embodiment, before performing verification operations on the first backup database and the second backup database respectively based on the checksum algorithm, the method further includes: detecting whether the status of the first backup database and the second backup database is normal; if the status of the first backup database and / or the second backup database is detected to be abnormal, the verification operation is not performed, and abnormal information is generated.

[0013] In one embodiment, before performing verification operations on the first backup database and the second backup database respectively based on the checksum algorithm, the method further includes: when it is detected that the backup database is running a replication thread that performs a replication operation on the primary database, counting the running frequency of the replication thread; when it is detected that the running frequency is higher than or equal to a frequency threshold, pausing the replication thread of the backup database, so that the data of the first backup database and the second backup database remain static during the execution of the verification operation.

[0014] In one embodiment, before performing verification operations on the first backup database and the second backup database respectively based on the checksum algorithm, the method further includes: obtaining the metadata of the database; generating a configuration table of the database based on the metadata; determining the first backup database and the second backup database from the backup database based on the configuration table, so as to generate the reference code for the first backup database based on the verification operation, and to generate the checksum for the second backup database based on the verification operation.

[0015] According to another aspect of this disclosure, a database consistency verification apparatus is provided for implementing the database consistency verification method described in any of the above embodiments.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute a database consistency verification method of any one of the above via executing the executable instructions.

[0017] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements a database consistency verification method according to any one of the above.

[0018] The database consistency verification scheme provided in the embodiments of this disclosure sends verification commands to the first and second standby databases to trigger checksum calculations, obtaining a reference code for the first standby database and a checksum code for the second standby database. The consistency between the first and second standby databases is determined by comparing whether the reference code and the checksum code are consistent. Furthermore, the consistency between the primary and standby databases is determined based on the consistency between the first and second standby databases. By generating comparison objects between standby databases through checksum calculations, the intrusion into the database can be reduced, thereby reducing the probability of abnormal database operation. Moreover, by performing consistency verifications regularly, inconsistent data in MySQL logical replication scenarios can be detected in a timely manner for prompt handling, ensuring high availability and strong consistency of the database.

[0019] In addition, using two standby databases for consistency comparison ensures the normal operation of the primary database. Furthermore, while implementing consistency verification between the primary and standby databases, it also enables consistency verification among multiple standby databases, thus providing greater flexibility. Moreover, even if anomalies occur during replication between the primary and standby databases, the verification operation can still be executed normally, thereby ensuring the reliability of the verification operation.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A schematic diagram of the system architecture of a database consistency verification method according to an embodiment of this disclosure is shown;

[0023] Figure 2A flowchart illustrating a database consistency verification method according to an embodiment of this disclosure is shown;

[0024] Figure 3 A flowchart illustrating another database consistency verification method in an embodiment of this disclosure is shown;

[0025] Figure 4 A flowchart illustrating another database consistency verification method in an embodiment of this disclosure is shown;

[0026] Figure 5 A flowchart illustrating yet another database consistency verification method according to an embodiment of this disclosure is shown;

[0027] Figure 6 A flowchart illustrating yet another database consistency verification method according to an embodiment of this disclosure is shown;

[0028] Figure 7 A flowchart illustrating yet another database consistency verification method according to an embodiment of this disclosure is shown;

[0029] Figure 8 A flowchart illustrating yet another database consistency verification method according to an embodiment of this disclosure is shown;

[0030] Figure 9 A flowchart illustrating yet another database consistency verification method according to an embodiment of this disclosure is shown;

[0031] Figure 10 A schematic diagram of a database consistency verification device according to an embodiment of the present disclosure is shown;

[0032] Figure 11 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0034] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The solution provided in this application defines multiple standby databases as a first standby database for reference and a second standby database for verification. Verification commands are sent to both the first and second standby databases to trigger verification and calculation, resulting in a reference code corresponding to the first standby database and a verification code corresponding to the second standby database. Furthermore, the consistency between the first and second standby databases is detected by comparing the reference code and the verification code. Based on the consistency between the first and second standby databases, the consistency between the primary and standby databases is determined. By using a verification and calculation method to generate comparison objects between standby databases, the intrusion into the database is reduced, thereby lowering the probability of abnormal database operation. Moreover, by performing consistency checks periodically, inconsistent data in MySQL logical replication scenarios can be detected and addressed promptly, ensuring high availability and strong consistency of the database.

[0036] To facilitate understanding, the following is an explanation of several terms used in this application.

[0037] Database instance: The data stored in a database exists on disk as database files, and a database instance is needed to perform database operations on these files. Taking MySQL as an example, a database instance can be understood as a collection of background processes and memory structures.

[0038] Logical replication: Its core principle is that the logical master database parses the WAL logs in the Publication table into a certain format and sends them to the logical slave database. The logical slave database Subscription receives the parsed WAL logs and redoes them, thereby achieving table data synchronization.

[0039] Statement: Statement is an important interface in Java for performing database operations. It is used to send SQL statements to the database after a database connection has been established. A statement object is used to execute a simple SQL statement without parameters.

[0040] Metadata, also known as intermediary data or relay data, is data about data. It primarily describes data properties and supports functions such as indicating storage location, historical data, resource lookup, and file records. Metadata is a type of electronic catalog; to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby assisting in data retrieval.

[0041] For primary and standby databases, modifying a table in the primary database will also modify the tables in the standby database. Furthermore, the establishment of primary and standby databases is typically based on the following considerations:

[0042] Disaster recovery: If the primary database is no longer available and the backup database is located in a different location, the backup database can take over immediately without any recovery time.

[0043] Load balancing: The primary database handles CRUD operations, while the standby database handles queries, so that many query operations do not consume resources on the primary database.

[0044] The solutions provided in this application involve technologies such as database processing, and are specifically illustrated through the following embodiments.

[0045] Figure 1 The diagram shows the system structure of a database consistency verification method according to an embodiment of the present disclosure, including multiple terminals 120 and a server cluster 140.

[0046] Terminal 120 can be a mobile terminal such as a mobile phone, game console, tablet computer, e-book reader, smart glasses, MP4 (Moving Picture Experts Group Audio Layer IV) player, smart home device, AR (Augmented Reality) device, VR (Virtual Reality) device, etc. Alternatively, terminal 120 can also be a personal computer (PC), such as a laptop computer and a desktop computer, etc.

[0047] The terminal 120 may contain an application that provides a database consistency verification method.

[0048] Terminal 120 is connected to server cluster 140 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0049] Server cluster 140 may be a single server, a combination of several servers, a virtualization platform, or a cloud computing service center. Server cluster 140 provides background services for applications that perform database consistency checks. Optionally, server cluster 140 undertakes the primary computational work, and terminal 120 undertakes secondary computational work; alternatively, server cluster 140 undertakes secondary computational work, and terminal 120 undertakes the primary computational work; or, terminal 120 and server cluster 140 collaborate on computation using a distributed computing architecture.

[0050] In some alternative embodiments, server cluster 140 is used to store database consistency verification models and prediction methods, etc.

[0051] Optionally, the application clients installed on different terminals 120 may be the same, or the application clients installed on two terminals 120 may be clients of the same type of application on different control system platforms. Depending on the terminal platform, the specific form of the application client may also differ; for example, the application client may be a mobile client, a PC client, or a World Wide Web (WWW) client.

[0052] Those skilled in the art will understand that the number of terminals 120 described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number of terminals or the type of device.

[0053] Optionally, the system may also include a management device ( Figure 1 (Not shown), the management device is connected to the server cluster 140 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0054] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0055] The following will describe in more detail the steps of the database consistency verification method in this example embodiment, with reference to the accompanying drawings and embodiments.

[0056] Figure 2 This diagram illustrates a flowchart of a database consistency verification method according to an embodiment of this disclosure. The method provided in this embodiment can be executed by any electronic device with computing power, such as... Figure 1 Terminal 120 and / or server cluster 140 in the example. In the following illustration, terminal 120 is used as the execution subject.

[0057] like Figure 2 As shown, terminal 120 is specifically a first electronic device. The first electronic device and the second electronic device can perform data interaction. The first electronic device is the requesting end, and the second electronic device is the responding end. Terminal 120 executes a database consistency verification method. The database includes a primary database and a backup database of the primary database. The backup database includes a first backup database and at least one second backup database. The method includes the following steps:

[0058] Step S202: In response to the verification command, verification operations are performed on the first backup database and the second backup database respectively based on the checksum algorithm. Specifically, a reference code is generated for the first backup database based on the verification operation, and a verification code is generated for the second backup database based on the verification operation.

[0059] There can be one or more second backup databases. When there are multiple second backup databases, the verification operation is performed on multiple second backup databases in parallel.

[0060] Specifically, checksums are used to verify the sum of data items in a database. By comparing the data in two databases, they detect the consistency between the two databases. However, since checksum calculations affect DML (Data Manipulation Language) operations on the database tables and the server's input / output, the checksum operation is performed between the standby databases to prevent impacting the real-time business operations of the primary database.

[0061] In addition, the check is performed on the entire instance, the entire database, and all tables. The check process only sends query requests to the database and does not write any information.

[0062] The first backup database can be understood as the reference database, and the second backup database can be understood as the verification database. There is usually one first backup database and one or more second backup databases.

[0063] Step S204: Determine the consistency between the first backup database and the second backup database based on the comparison result of the reference code and the check code.

[0064] Step S206: Determine the consistency between the primary database and the backup database based on the consistency between the first backup database and the second backup database.

[0065] If the first standby database and the second standby database are inconsistent, it can be confirmed that there is an abnormal standby database, and thus it can be determined that the standby database and the primary database are definitely inconsistent. Since the standby database is modified along with the primary database, if the first standby database and the second standby database are consistent, it can be considered that the primary database and the standby database are consistent. Thus, consistency detection between the primary and standby databases is achieved based on consistency detection between standby databases.

[0066] In this embodiment, multiple standby databases are defined as a first standby database as a reference object and a second standby database as a verification object. Verification commands are sent to the first and second standby databases to trigger them to perform verification and calculation, obtaining a reference code corresponding to the first standby database and a verification code corresponding to the second standby database. Furthermore, the consistency between the first and second standby databases is detected by comparing whether the reference code and the verification code are consistent. The consistency between the primary and standby databases is then determined based on the consistency between the first and second standby databases. By using a verification and calculation method to generate comparison objects between standby databases, the intrusion into the database can be reduced, thereby reducing the probability of abnormal database operation. Moreover, by performing consistency checks regularly, inconsistent data in MySQL logical replication scenarios can be detected in a timely manner for prompt handling, ensuring high availability and strong consistency of the database.

[0067] In addition, using two standby databases for consistency comparison ensures the normal operation of the primary database. Furthermore, while implementing consistency verification between the primary and standby databases, it also enables consistency verification among multiple standby databases, thus providing greater flexibility. Moreover, even if anomalies occur during replication between the primary and standby databases, the verification operation can still be executed normally, thereby ensuring the reliability of the verification operation.

[0068] like Figure 3 As shown, in one embodiment, step S202 includes an initial verification operation, which performs verification operations on the first backup database and the second backup database based on a checksum algorithm, specifically including:

[0069] Step S302: When performing the first verification operation on the first backup database, traverse all the first data tables in the first backup database and calculate the total verification of all the first data tables in the first backup database as the initial reference code.

[0070] Step S304: When performing the first verification operation on the second backup database, traverse all the second data tables in the second backup database and calculate the total verification of all the second data tables in the second backup database as the initial verification code.

[0071] As a specific method for calculating checksums, it includes:

[0072] First, set the checksum of the data to be checked to 0. Then, divide the data into 2-byte segments, forming a 16-bit value for each 2-byte segment. If there is a single byte of data at the end, add a byte of 0 to form a 2-byte segment. Next, accumulate all the 16-bit values ​​into a 32-bit value. Then, add the high 16 bits and low 16 bits of the 32-bit value into a new 32-bit value. If the new 32-bit value is greater than 0xffff, add the high 16 bits and low 16 bits of the new value again. Finally, invert the 16-bit value obtained in the previous step to get the checksum value, which can then be stored in the checksum field of the data.

[0073] In this embodiment, when performing the verification operation for the first time, all data tables in the backup database are traversed and verified to calculate the verification sum of the first backup database as the initial reference code, and the verification sum of the second backup database is calculated as the initial verification code. The consistency result is obtained by further comparing the initial reference code and the initial verification code. By calculating the verification sum for all first data tables and all second data tables respectively, the reliability of the consistency comparison process of multiple backup databases is ensured.

[0074] In addition, during the verification process, multiple backup databases, as well as multiple tables in each backup database, can be executed concurrently and run independently, which can greatly improve the efficiency of the entire verification process.

[0075] like Figure 4 As shown, in one embodiment, the verification operation further includes a second verification operation, which performs verification operations on the first backup database and the second backup database respectively based on the checksum algorithm, specifically including:

[0076] Step S402: When the initial reference code and the initial check code are found to be inconsistent, query the bit value fields that are inconsistent between the initial reference code and the initial check code based on the comparison result.

[0077] Step S404: Based on the inconsistent bit value fields, determine the reference data tables that need to be re-verified in all first data tables, and the verification data tables that need to be re-verified in all second data tables.

[0078] Step S406: Perform a second verification operation on the reference data table and the verification data table.

[0079] The re-verification operation can be performed in one or more rounds.

[0080] In this embodiment, by setting up multiple rounds of verification operations, a full check of all tables is achieved by calculating the total verification of all tables during the first verification operation. When an inconsistency is detected between the initial reference code and the initial check code, the inconsistent bit value fields in the initial reference code and the initial check code are queried based on the comparison results to find the inconsistent set of tables, namely the corresponding reference data table and verification data table. During subsequent verification operations, the reference data table and the verification data table are checked separately, which reduces the workload of the verification operation while ensuring the reliability of the verification.

[0081] like Figure 5 As shown, in one embodiment, step S406 involves performing a re-verification operation on the reference data table and the verification data table, specifically including:

[0082] Step S502: Update the checksum of the reference data table based on the preset update frequency, and use it as the update reference code.

[0083] Step S504: Update the total verification value of the verification data table based on the update frequency, and use it as the update verification code.

[0084] Specifically, the update operations for the reference data table and the verification data table involve recalculating the checksum for the reference data table and recalculating the checksum for the verification data table.

[0085] In step S204, the consistency between the first backup database and the second backup database is determined based on the comparison result of the reference code and the check code, specifically including:

[0086] Step S506: After each update is completed, compare the update reference code with the update check code.

[0087] Step S508: When the number of updates is less than the threshold for the first update and the update reference code and update check code are found to be consistent, the first backup database and the second backup database are determined to be consistent.

[0088] Step S510: When the number of updates is detected to be greater than or equal to the first number threshold and less than or equal to the second number threshold, the update reference code and the update check code are still inconsistent. It is determined that the first backup database and the second backup database are inconsistent, and the second number threshold is greater than or equal to the first number threshold.

[0089] The first threshold represents the maximum number of times a detection result showing that the updated reference code and the updated check code are consistent is valid, while the second threshold represents the number of times the check operation is stopped.

[0090] In this embodiment, during the consistency verification process, since there is a situation where the standby database synchronizes the data with the primary database, that is, some data tables in the standby database are written to in real time by business operations, there is a possibility of false results in the checksum check. By pre-setting the number of verifications, multiple rounds of checks are implemented. The first time, all tables are checked in full to find the set of inconsistent tables. The second time, the inconsistent tables are checked separately. If they are still inconsistent, a second retry can be performed to reduce false alarms and ensure the accuracy of the verification results.

[0091] like Figure 6 As shown, in one embodiment, before performing verification operations on the first and second backup databases respectively based on the checksum algorithm, the method further includes:

[0092] Step S602: Check whether the status of the first backup database and the second backup database is normal.

[0093] Step S604: When an abnormal status is detected in the first backup database and / or the second backup database, no verification operation is performed, and abnormal information is generated.

[0094] In this embodiment, the status of the first and second backup databases is checked to see if they are normal. Specifically, the status of the first and second backup databases is checked to see if they are in a read-only state. If they are in a read-only state, the backup databases are considered to be in a normal state. In this case, only the first and second backup databases are allowed to copy data from the primary database.

[0095] In this embodiment, by configuring the first and second backup databases to read-only mode, it is possible to prevent database crashes or switching anomalies during the verification process. This ensures the reliability of database operation during the verification process while also guaranteeing the validity of the verification results.

[0096] Otherwise, if an abnormal status is detected in the first backup database and / or the second backup database, the current verification process will be terminated and the error information will be reported and entered into the database.

[0097] like Figure 7 As shown, in one embodiment, before performing verification operations on the first and second backup databases respectively based on the checksum algorithm, the method further includes:

[0098] Step S702: When it is detected that the standby database is running a replication thread that performs a replication operation on the primary database, the running frequency of the replication thread is counted.

[0099] Step S704: When the running frequency is detected to be higher than or equal to the frequency threshold, the replication thread of the standby database is paused so that the data of the first standby database and the second standby database remain static during the verification operation.

[0100] In this embodiment, for scenarios with frequent writes, the replication thread between the primary and secondary databases is paused before the verification operation is performed, so that the first and second secondary databases stop on the same node, thereby ensuring the accuracy of the verification results by comparing static data.

[0101] like Figure 8 As shown, in one embodiment, before performing verification operations on the first and second backup databases respectively based on the checksum algorithm in step S202, the method further includes:

[0102] Step S 802: Obtain the metadata of the database.

[0103] Step S 804: Generate the database configuration table based on metadata.

[0104] Step S 806: Based on the configuration table, determine the first backup database and the second backup database from the backup database, generate a reference code for the first backup database based on the verification operation, and generate a verification code for the second backup database based on the verification operation.

[0105] In this embodiment, for a database cluster with a primary-multiple backup or cross-data center multi-node deployment architecture, it is necessary to determine the reference database and the verification database before performing the consistency verification operation. That is, the first backup database is configured as the reference database, the second backup database is configured as the verification database, and the metadata information of each database cluster is entered into the configuration table to ensure that the verification operation verifies each node on a database-by-database basis.

[0106] Furthermore, by determining the number of second backup databases, the verification and calculation of multiple second backup databases can be reliably executed concurrently.

[0107] In one embodiment, the method further includes: generating a result record table based on the comparison result and the configuration table; and storing the result record table in a specified storage area, which is located outside the primary database, the first standby database, and the second standby database.

[0108] In this embodiment, the current primary database or main data center information is determined based on the metadata information in the configuration table. The first standby database is selected as the reference database instance for inspection, and the other second standby databases are used as verification databases. The verification databases are traversed sequentially based on the reference database instance. After each group of inspections is completed, the inspection information is written to the result record table. By generating the result record table separately, no write operation is performed on the database, which helps to further reduce the intrusion on the data.

[0109] In one embodiment, generating a result record table based on the comparison result further includes: when the reference code and the check code are inconsistent, determining the inconsistent bit value fields in the reference code and the check code; determining the inconsistent first data table in the first backup database and the second data table in the second backup database based on the bit value fields, and writing the identifiers of the first data table and the second data table into the result record table.

[0110] In this embodiment, if the verification results are consistent, the verification passes; otherwise, the inconsistent table set is returned and written to the result record table to ensure accurate location of the anomaly, thereby improving the efficiency of anomaly repair.

[0111] like Figure 9 As shown, another database consistency verification method according to an embodiment of this disclosure includes:

[0112] Step S902: Obtain the database metadata.

[0113] Step S904: Generate a database configuration table based on metadata.

[0114] Step S906: Determine the first and second standby databases from the standby databases based on the configuration table.

[0115] In step S908, in response to the verification command, check whether the status of the first backup database and the second backup database is normal. If "yes", proceed to step S912; if "no", proceed to step S910.

[0116] In step S910, no verification operation is performed, and an exception message is generated.

[0117] Step S912: When it is detected that the standby database is running a replication thread that performs a replication operation on the primary database, the running frequency of the replication thread is counted.

[0118] Step S914: When the running frequency is detected to be higher than or equal to the frequency threshold, the replication thread of the standby database is paused so that the data of the first standby database and the second standby database remain static during the verification operation.

[0119] Step S916: When performing the first verification operation on the first backup database, all first data tables in the first backup database are traversed, and the total verification sum of all first data tables in the first backup database is calculated as the initial reference code. When performing the first verification operation on the second backup database, all second data tables in the second backup database are traversed, and the total verification sum of all second data tables in the second backup database is calculated as the initial verification code.

[0120] Step S918: When the initial reference code and the initial check code are found to be inconsistent, query the bit value fields that are inconsistent between the initial reference code and the initial check code based on the comparison result.

[0121] Step S920: Based on the inconsistent bit value fields, determine the reference data tables that need to be re-verified in all first data tables, and the verification data tables that need to be re-verified in all second data tables.

[0122] Step S922: Update the total verification value of the reference data table based on the preset update frequency, and use it as the update reference code; update the total verification value of the verification data table based on the update frequency, and use it as the update verification code.

[0123] Step S924: After each update is completed, compare the update reference code with the update check code.

[0124] Step S926: When the number of updates is less than the threshold for the first update and the update reference code and update check code are found to be consistent, determine that the first backup database and the second backup database are consistent.

[0125] Step S928: When the number of updates is detected to be greater than or equal to the first number threshold and less than or equal to the second number threshold, and the update reference code and update check code are still inconsistent, it is determined that the first backup database and the second backup database are inconsistent.

[0126] Among them, the threshold for the second number is greater than or equal to the threshold for the first number.

[0127] Step S930: Determine the consistency between the primary database and the backup database based on the consistency between the first backup database and the second backup database.

[0128] In this embodiment, since checksums affect DML (Data Manipulation Language) operations on tables and server I / O read / write, the check scheme selects to perform the check between at least two standby databases to avoid affecting the real-time business of the primary database.

[0129] For a multi-node deployment architecture with one primary database and multiple backup databases, or a multi-node deployment architecture across data centers, it is necessary to determine the reference instance and the instance to be inspected before inspection. Based on the configuration database metadata information, determine the current primary database or main data center information, select a backup database as the reference or baseline instance for inspection, and other non-primary databases as the set of objects to be inspected. Using the reference instance as the baseline, iterate through the set of instances to be inspected in sequence. After each group is inspected, the inspection information is written to the inspection result table.

[0130] Furthermore, since the verification is performed on the entire instance, the entire database, and all tables, only query requests are sent to the database during the verification process, and no information is written. If the verification results are consistent, the verification passes; otherwise, a set of inconsistent tables is returned and written to the check result table.

[0131] During the consistency comparison process, the standby database is still synchronizing data with the primary database. At this time, some tables may be experiencing real-time writes, potentially leading to inconsistent checksum results. A multi-round check can be employed: the first round performs a full check on all tables to identify the inconsistent set; the second round checks the inconsistent tables individually; and if inconsistencies persist, a retry can be performed to reduce false positives. For scenarios with excessively frequent writes, the replication thread can be stopped during the check, allowing both standby databases to remain in the same position. Comparing static data in this case ensures the accuracy of the check results.

[0132] To prevent abnormal scenarios such as database crashes or switching during the inspection process, each inspection must confirm that the reference database and the database being inspected are both normal and read-only. Otherwise, the inspection should be terminated and the error information should be reported to the database.

[0133] Throughout the verification process, multiple tables can be executed concurrently and run independently, greatly improving the efficiency of the entire verification process.

[0134] Through the above verification process, the verification based on the consistency of data in the backup database across multiple nodes reflects the consistency of data between the primary and backup databases. Furthermore, the database check process does not involve any intrusion into the database and has minimal dependence on the database itself.

[0135] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0136] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0137] The following reference Figure 10 This describes a database consistency verification device 1000 according to this embodiment of the present invention. Figure 10 The database consistency verification device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0138] The database consistency verification device 1000 is manifested as a hardware module. Components of the database consistency verification device 1000 may include, but are not limited to: a verification module 1002, used to perform verification operations on a first backup database and a second backup database respectively based on a checksum algorithm in response to a verification command, wherein a reference code is generated for the first backup database based on the verification operation, and a check code is generated for the second backup database based on the verification operation; a first determination module 1004, used to determine the consistency between the first backup database and the second backup database based on the comparison result of the reference code and the check code; and a second determination module 1006, used to determine the consistency between the primary database and the backup database based on the consistency between the first backup database and the second backup database.

[0139] In one embodiment, the verification module 1002 is further configured to: when performing the first verification operation on the first backup database, traverse all first data tables in the first backup database and calculate the total verification of all first data tables in the first backup database as an initial reference code; when performing the first verification operation on the second backup database, traverse all second data tables in the second backup database and calculate the total verification of all second data tables in the second backup database as an initial verification code.

[0140] In one embodiment, the verification module 1002 is further configured to: when an inconsistency is detected between the initial reference code and the initial verification code, query the inconsistent bit value fields in the initial reference code and the initial verification code based on the comparison result; determine the reference data tables that need to be verified again in all first data tables and the verification data tables that need to be verified again in all second data tables based on the inconsistent bit value fields; and perform a re-verification operation on the reference data tables and the verification data tables.

[0141] In one embodiment, the verification module 1002 is further configured to: update the verification sum of the reference data table based on a preset update frequency, as an update reference code; and update the verification sum of the verification data table based on the update frequency, as an update verification code; the first determination module 1004 is further configured to: compare the update reference code with the update verification code after each update; when the number of updates is less than the first number threshold and the update reference code and update verification code are detected to be consistent, determine that the first backup database and the second backup database are consistent; when the number of updates is detected to be greater than or equal to the first number threshold and less than or equal to the second number threshold, and the update reference code and update verification code are still inconsistent, determine that the first backup database and the second backup database are inconsistent, and the second number threshold is greater than or equal to the first number threshold.

[0142] In one embodiment, the system further includes: a detection module 1008, used to detect whether the status of the first backup database and the second backup database is normal; when the status of the first backup database and / or the second backup database is detected to be abnormal, no verification operation is performed, and abnormal information is generated.

[0143] In one embodiment, the system further includes: a statistics module 1010, configured to count the running frequency of a replication thread when a replication thread performing a replication operation on the primary database is detected in the standby database; and to pause the replication thread of the standby database when the running frequency is detected to be higher than or equal to a frequency threshold, so that the data of the first standby database and the second standby database remain static during the verification operation.

[0144] In one embodiment, the system further includes: an acquisition module 1002, configured to acquire metadata of the database; generate a configuration table of the database based on the metadata; determine a first backup database and a second backup database from the backup database based on the configuration table, and generate a reference code for the first backup database based on a verification operation, and generate a verification code for the second backup database based on a verification operation.

[0145] The following reference Figure 11 To describe an electronic device 1100 according to this embodiment of the present invention. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0146] like Figure 11 As shown, the electronic device 1100 is manifested in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, and a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110).

[0147] The storage unit stores program code, which can be executed by the processing unit 1110, causing the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1010 can perform, as follows: Figure 2 Steps S202 to S206 shown, and other steps defined in the database consistency verification method of this disclosure.

[0148] Storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 11201 and / or cache memory 11202, and may further include a read-only memory (ROM) 11203.

[0149] Storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0150] Bus 1130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0151] Electronic device 1100 can also communicate with one or more external devices 1160 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1150. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1150. As shown, network adapter 1150 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0152] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0153] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0154] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0155] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0156] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0157] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0158] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0159] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0160] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0161] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A database consistency verification method, characterized in that, The database includes a primary database and a backup database of the primary database, the backup database including a first backup database and at least one second backup database, and the method includes: In response to the verification command, when performing the first verification operation on the first backup database, all first data tables in the first backup database are traversed, and the total verification of all first data tables in the first backup database is calculated as the initial reference code. When performing the initial verification operation on the second backup database, all second data tables in the second backup database are traversed, and the total verification value of all second data tables in the second backup database is calculated as the initial verification code. The consistency between the first backup database and the second backup database is determined based on the comparison result between the initial reference code and the initial verification code. When the initial reference code is found to be inconsistent with the initial check code, the bit value fields that are inconsistent between the initial reference code and the initial check code are queried based on the comparison result; Based on the inconsistent bit value field, determine all reference data tables in the first data table that need to be re-verified, and all verification data tables in the second data table that need to be re-verified; The total verification value of the reference data table is updated based on a preset update frequency to serve as the update reference code; and the total verification value of the verification data table is updated based on the update frequency to serve as the update verification code; wherein, a reference code is generated for the first backup database based on the verification operation, and a verification code is generated for the second backup database based on the verification operation. The consistency between the first backup database and the second backup database is determined based on the comparison result of the update reference code and the update verification code. The consistency between the primary database and the backup database is determined based on the consistency between the first backup database and the second backup database. If the first backup database and the second backup database are inconsistent, it is confirmed that there is an abnormal backup database, and it is determined that the backup database and the primary database are definitely inconsistent. If the first backup database and the second backup database are consistent, it is determined that the primary database and the backup database are consistent.

2. The database consistency verification method according to claim 1, characterized in that, The step of determining the consistency between the first backup database and the second backup database based on the comparison result of the update reference code and the update check code specifically includes: After each update is completed, the update reference code is compared with the update verification code; When the number of updates is less than the threshold for the first update, and the update reference code is found to be consistent with the update check code, it is determined that the first backup database and the second backup database are consistent. When the number of updates is detected to be greater than or equal to the first number threshold and less than or equal to the second number threshold, and the update reference code and the update check code are still inconsistent, it is determined that the first backup database and the second backup database are inconsistent, and the second number threshold is greater than or equal to the first number threshold.

3. The database consistency verification method according to claim 1, characterized in that, Before performing verification operations on the first backup database and the second backup database respectively based on the checksum algorithm, the method further includes: Check whether the status of the first backup database and the second backup database is normal; If an abnormality is detected in the status of the first backup database and / or the second backup database, the verification operation is not performed, and an error message is generated.

4. The database consistency verification method according to claim 1, characterized in that, Before performing verification operations on the first backup database and the second backup database respectively based on the checksum algorithm, the method further includes: When it is detected that the standby database is running a replication thread that performs a replication operation on the primary database, the running frequency of the replication thread is counted. When the frequency of operation is detected to be higher than or equal to the frequency threshold, the replication thread of the backup database is paused so that the data of the first backup database and the second backup database remain static during the execution of the verification operation.

5. The database consistency verification method according to any one of claims 1 to 4, characterized in that, Before performing verification operations on the first backup database and the second backup database respectively based on the checksum algorithm, the method further includes: Obtain the metadata of the database; The configuration table of the database is generated based on the metadata; Based on the configuration table, the first backup database and the second backup database are determined from the backup database, so as to generate the reference code for the first backup database based on the verification operation, and to generate the verification code for the second backup database based on the verification operation.

6. A database consistency verification device, characterized in that, This is used to implement the database consistency verification method according to any one of claims 1 to 5.

7. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the database consistency verification method according to any one of claims 1 to 5 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the database consistency verification method according to any one of claims 1 to 5.

Images