Data verification method, related device, equipment and storage medium

Abstract

The invention discloses a data verification method, a related device, equipment and a storage medium. The method comprises the steps of obtaining a basic line number of a first data table in a target end database at a first moment; obtaining first statistical information of a second data table in the source end database from a first moment to a second moment, wherein the first statistical information comprises a first newly-added line number and a first deleted line number which are obtained based on log data statistics of the source end database; acquiring a real-time line number of the first data table at a second moment; according to the basic line number at the first moment, the first newly-added line number and the first deleted line number, determining the statistical line number of the first data table at the second moment; and performing consistency verification on the real-time row number of the first data table at the second moment and the statistical row number of the first data table at the second moment to obtain a first verification result of the first data table. According to the method and the device, the data verification task can be efficiently and quickly realized under the condition that a source end database does not need to be inquired and a source end data table structure does not need to be modified.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, related apparatus, device, and storage medium for data verification. Background Technology

[0002] As database usage scenarios become increasingly diverse, the resulting database usage requirements are also becoming more complex. In order to comprehensively utilize data of different values, a hybrid transactional and analytical processing (HTAP) database system is proposed to bridge the gap between online transaction processing (OLTP) and online analytical processing (OLAP).

[0003] In loosely coupled HTAP database systems, the data consistency issue needs to be addressed when migrating from OLTP to OLAP databases. Currently, table checksum tools (pt-table-checksum) are primarily used to verify data consistency. This involves first calculating a hash checksum for each row of data in both the source and target tables, then comparing the checksums to determine if discrepancies exist, and finally providing a detailed replication consistency report.

[0004] In the process of researching and practicing current technologies, the inventors discovered that the current solutions have at least the following problems: pt-table-checksum is applicable only under homogeneous storage and with consistent original data. However, row-based and column-based storage in loosely coupled HTAP database systems exhibit heterogeneity in storage format, preprocessing logic, and synchronization mechanisms, making pt-table-checksum unsuitable. Therefore, an effective method is urgently needed to address these issues. Summary of the Invention

[0005] This application provides a data verification method, related apparatus, device, and storage medium. Without querying the source database or modifying the source data table structure, it does not increase the load on the source database, thus enabling efficient and rapid data verification tasks.

[0006] In view of this, this application provides a data verification method, including:

[0007] Obtain the number of basic rows in the first data table of the target database at the first moment;

[0008] Obtain first statistical information of the second data table in the source database from the first time point to the second time point. The second data table is a heterogeneous data table or a homogeneous data table corresponding to the first data table. The second time point occurs after the first time point. The first statistical information includes the first number of newly added rows and the first number of deleted rows obtained from the log data statistics of the source database.

[0009] Obtain the real-time row count of the first data table at the second time point;

[0010] Based on the number of basic rows, the number of newly added rows, and the number of deleted rows at the first moment, determine the number of statistical rows in the first data table at the second moment;

[0011] The consistency of the real-time row count and the statistical row count of the first data table at the second time point is checked to obtain the first check result of the first data table.

[0012] Another aspect of this application provides a data verification device, comprising:

[0013] The acquisition module is used to obtain the basic row count of the first data table in the target database at the first moment.

[0014] The acquisition module is also used to acquire first statistical information of the second data table in the source database from the first time to the second time, wherein the second data table is a heterogeneous data table or a homogeneous data table corresponding to the first data table, the second time occurs after the first time, and the first statistical information includes the first number of newly added rows and the first number of deleted rows obtained from the log data statistics of the source database.

[0015] The acquisition module is also used to acquire the real-time row count of the first data table at the second moment;

[0016] The determination module is used to determine the number of statistical rows in the first data table at the second time based on the basic number of rows, the first number of newly added rows, and the first number of deleted rows at the first time.

[0017] The verification module is used to verify the consistency between the real-time row count and the statistical row count of the first data table at the second time and obtain the first verification result of the first data table.

[0018] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0019] The acquisition module is specifically used to send a first query statement to the target database, wherein the first query statement includes the database name of the target database and the table name corresponding to the first data table;

[0020] Receive the first query result returned by the target server, wherein the first query result includes the basic row count of the first data table at the first moment, and the first moment is the moment when the target database responds to the first query statement by performing a row count query on the first data table.

[0021] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0022] The acquisition module is specifically used to send a second query statement to the target database, wherein the second query statement includes the database name of the target database and the table name corresponding to the first data table;

[0023] Receive the second query result returned by the target server, wherein the second query result includes the real-time row count of the first data table at the second moment, and the second moment is the moment when the target database responds to the second query statement by querying the row count of the first data table.

[0024] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0025] The acquisition module is specifically used to acquire statistical information corresponding to each checkpoint from the first time to the second time for the second data table in the source database. The checkpoint is used to indicate the time when the log data of the source database is statistically analyzed. The incremental log data is the log data within the check time period. The statistical information includes the number of new rows and the number of deleted rows in the second data table within the check time period.

[0026] The number of newly added rows in the statistics corresponding to each checkpoint is summed to obtain the first number of newly added rows in the second data table from the first time point to the second time point;

[0027] The number of deleted rows in the statistics corresponding to each checkpoint is summed to obtain the first number of deleted rows in the second data table from the first time point to the second time point.

[0028] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0029] The determination module is specifically used to sum the base number of rows and the first newly added row number at the first moment to obtain the incremental number of rows;

[0030] The difference between the incremental row count and the first deleted row count is used to obtain the statistical row count of the first data table at the second time step.

[0031] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0032] The determination module is specifically used to determine the basic row count at the first moment as the statistical row count of the first data table at the second moment when the first number of newly added rows is equal to the first number of deleted rows.

[0033] If the number of newly added rows is not equal to the number of newly deleted rows, the number of basic rows at the first time step is summed with the number of newly added rows to obtain the incremental row count. The difference between the incremental row count and the number of deleted rows is then calculated to obtain the statistical row count of the first data table at the second time step.

[0034] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0035] The acquisition module is also used to acquire the basic row count of the first data table at the second time step;

[0036] The acquisition module is also used to acquire second statistical information of the second data table from the second time to the third time, wherein the third time occurs after the second time, and the duration from the second time to the third time is equal to the duration from the first time to the second time. The second statistical information includes the second number of newly added rows and the second number of deleted rows obtained from the log data statistics of the source database.

[0037] The acquisition module is also used to acquire the real-time row count of the first data table at the third moment;

[0038] The determination module is also used to determine the number of rows in the first data table at the third time based on the basic row count, the second number of newly added rows, and the second number of deleted rows at the second time step.

[0039] The verification module is also used to verify the consistency between the real-time row count of the first data table at the third time and the statistical row count at the third time, and to obtain the second verification result of the first data table.

[0040] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0041] The acquisition module is specifically used to determine the second moment in the event of a fault detection;

[0042] Obtain the first statistical information of the second data table in the source database from the first time point to the second time point;

[0043] The acquisition module is also used to acquire the basic row count of the first data table at the fourth time point after the fault is recovered;

[0044] The acquisition module is also used to acquire the third statistical information of the second data table from the fourth time to the fifth time, wherein the fifth time occurs after the fourth time, and the duration from the fourth time to the fifth time is longer than the duration from the first time to the second time. The third statistical information includes the third number of newly added rows and the third number of deleted rows obtained from the log data statistics of the source database.

[0045] The acquisition module is also used to acquire the real-time row count of the first data table at the fifth moment;

[0046] The determination module is also used to determine the number of rows in the first data table at the fifth time step based on the basic row count at the fourth time step, the third number of newly added rows, and the third number of deleted rows.

[0047] The verification module is also used to verify the consistency between the real-time row count and the statistical row count of the first data table at the fifth time and obtain the third verification result of the first data table.

[0048] In one possible design, in another implementation of another aspect of the embodiments of this application, the data verification device further includes a storage module;

[0049] The acquisition module is also used to acquire statistical information corresponding to the checkpoints, including the number of updated rows in the second data table during the inspection period.

[0050] The storage module is used to determine the storage space of the second data table based on the number of updated rows in the second data table during each inspection time period, provided that the storage conversion conditions are met. The storage space can be either hot storage space or cold storage space.

[0051] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0052] The acquisition module is also used to acquire statistical information corresponding to the checkpoints. The statistical information also includes the baseline refresh identifier field of the second data table during the inspection period.

[0053] The determination module is also used to determine the time corresponding to the checkpoint as the first time for data consistency verification when the baseline refresh identifier field included in the statistics indicates that the overlapping phase of full data migration and incremental data migration has ended.

[0054] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0055] The acquisition module is also used to acquire statistical information corresponding to the checkpoint closest to the third time from the second time to the third time when the first verification result of the first data table indicates that the data verification is consistent and the second verification result of the first data table indicates that the data verification is inconsistent. The statistical information also includes the update time of the second data table.

[0056] The determination module is also used to determine the target time period in which the synchronization between the first data table and the second data table fails, based on the second time period and the update time included in the statistical information. The start time of the target time period is the second time period, and the end time of the target time period is the update time included in the statistical information.

[0057] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0058] The acquisition module is also used to acquire statistical information corresponding to the checkpoints. The statistical information includes the time when the first data table and the second data table last matched.

[0059] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0060] The acquisition module is also used to acquire statistical information corresponding to the checkpoints. The statistical information includes log positions that are consistent in the most recent verification between the first data table and the second data table. The log positions are used to identify log events in the log data.

[0061] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0062] The verification module is specifically used to obtain the statistical information of the next checkpoint if the consistency verification between the real-time row count and the statistical row count of the first data table at the second time point is not completed within the verification timeout period.

[0063] Based on the basic row count at the first moment and the statistical information at the next checkpoint, determine the number of statistical rows in the first data table at the next checkpoint;

[0064] Get the real-time row count of the first data table at the next checkpoint;

[0065] If the consistency verification between the real-time row count and the statistical row count of the first data table at the next checkpoint is completed within the verification timeout period, the first verification result of the first data table is obtained.

[0066] In one possible design, in another implementation of another aspect of the embodiments of this application, the data verification device further includes an execution module;

[0067] The execution module is used to perform a consistency check on the real-time row count and the statistical row count of the first data table at the second time. After obtaining the first check result of the first data table, if the data check result indicates that the data is inconsistent, the module executes the operation corresponding to the error mode identifier. The operation corresponding to the error mode identifier includes at least one of the following:

[0068] Generate alarm notification messages;

[0069] Interrupt the synchronization between the first and second data tables;

[0070] Stop performing data consistency checks.

[0071] In another aspect, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the methods described above.

[0072] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described above.

[0073] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0074] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0075] This application provides a data verification method. It requires obtaining the basic row count of a first data table in the target database at a first moment and the real-time row count at a second moment. It also requires obtaining the first statistical information (i.e., the first number of newly added rows and the first number of deleted rows) of a second data table in the source database from the first moment to the second moment. Based on this, the statistical row count of the first data table at the second moment is determined according to the basic row count, the first number of newly added rows, and the first number of deleted rows at the first moment. Then, a consistency check is performed between the real-time row count and the statistical row count of the first data table at the second moment to obtain the first verification result of the first data table. Through the above method, log data from the source database over a period of time is extracted using a data synchronization component, and the number of newly added rows and deleted rows in the source data table (i.e., the second data table) during this period is statistically obtained based on this log data. This is used as the basis for calculating the statistical row count of the target data table (i.e., the first data table). Data verification is completed by comparing the consistency between the statistical row count and the real-time row count of the target data table. Therefore, this application does not increase the load on the source database without querying the source database or modifying the source data table structure, thus enabling efficient and fast data verification tasks. Attached Figure Description

[0076] Figure 1 This is a schematic diagram of the architecture of an HTAP database system in an embodiment of this application;

[0077] Figure 2 This is a schematic diagram of a data table synchronization scenario in an embodiment of this application;

[0078] Figure 3 This is another schematic diagram illustrating a data table synchronization scenario in an embodiment of this application;

[0079] Figure 4 This is another schematic diagram illustrating a data table synchronization scenario in an embodiment of this application;

[0080] Figure 5 This is a schematic diagram of an implementation environment for the data verification method in this application.

[0081] Figure 6 This is a flowchart illustrating a data verification method in an embodiment of this application;

[0082] Figure 7 This is a schematic diagram illustrating the consistency check of a data table in an embodiment of this application;

[0083] Figure 8 This is a schematic diagram illustrating statistical information obtained based on checkpoint aggregation in an embodiment of this application;

[0084] Figure 9 This is a schematic diagram illustrating data consistency verification based on multiple verification cycles in an embodiment of this application;

[0085] Figure 10 This is a schematic diagram illustrating a failure that occurs during the data writing process in an embodiment of this application;

[0086] Figure 11 This is another schematic diagram illustrating a failure that occurred during the data writing process in an embodiment of this application;

[0087] Figure 12 This is a schematic diagram of the latest log position corresponding to the source database in an embodiment of this application;

[0088] Figure 13 This is a schematic diagram of a process for starting incremental data migration in an embodiment of this application;

[0089] Figure 14 This is a schematic diagram illustrating the overlap between full data migration and incremental data migration in an embodiment of this application;

[0090] Figure 15 This is a schematic diagram of the overall process of the data verification method in the embodiments of this application;

[0091] Figure 16 This is a schematic diagram of a data verification device in an embodiment of this application;

[0092] Figure 17 This is a schematic diagram of the structure of a computer device in an embodiment of this application. Detailed Implementation

[0093] This application provides a data verification method, related apparatus, device, and storage medium. Without querying the source database or modifying the source data table structure, it does not increase the load on the source database, thus enabling efficient and rapid data verification tasks.

[0094] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0095] In database management systems (DBMS), OLTP and OLAP databases are typically separated. Ideally, a DBMS should possess both the real-time transaction processing capabilities of an OLTP database and the ability to handle complex queries in an OLAP database. Such a database is often referred to as an HTAP database. Loosely coupled HTAP is a common HTAP database architecture. It imports data from an OLTP database to an OLAP database using extract-transform-load (ETL) tools. However, ETL tools often have significant latency and cannot capture changes in the source data in a timely manner. To address the problems of ETL, high-speed link synchronization components based on log-based master-slave replication (i.e., change data capture (CDC) components) have become a more efficient and stable data synchronization tool.

[0096] For easier understanding, please refer to Figure 1 , Figure 1This is a schematic diagram of the architecture of the HTAP database system in this application embodiment. As shown in the figure, the OLAP database is equivalent to a special read-only copy of the OLTP database. Large amounts of data stored by users in the OLTP database are synchronized to the OLAP database through a high-speed link synchronization component. When users execute different query statements, the database's front-end agent engine analyzes the execution statements and forwards the most advantageous execution statements from the OLTP database (e.g., transactional statements with a high proportion of insert, delete, and update operations) to the corresponding OLTP nodes for execution. Conversely, it forwards the most advantageous execution statements from the OLAP database (e.g., large queries, complex queries, etc.) to the corresponding OLAP nodes for execution.

[0097] HTAP databases need to support both OLTP and OLTP databases, but their data access patterns and requirements differ. Data consistency verification ensures that the data used by transactional operations and analytical queries remains consistent, preventing business decision-making biases or transaction processing errors. Currently, pt-table-checksum is commonly used for data consistency verification. The core principle of pt-table-checksum is to perform checksum calculations on the source database in a block-by-block manner, based on replace-into statements, using cyclic redundancy check 32-bit (CRC32) or message-digest algorithm 5 (MD5) to generate a checksum for each block of data. These calculations are performed with low priority and automatically synchronized to the target database through a replication mechanism. After receiving the checksum in Structured Query Language (SQL), the target database repeats the same calculation process locally. pt-table-checksum then identifies differences by comparing the results on the source and target databases, ultimately providing a detailed replication consistency report.

[0098] However, pt-table-checksum cannot be directly used for data consistency verification in loosely coupled HTAP database systems. The core obstacle lies in the fact that the source and target databases use different underlying storage engines and data organization formats. The core principle of pt-table-checksum is to calculate the CRC32 or MD5 checksum of data blocks in both databases by executing the same SQL query for comparison. However, row-based and column-based storage may differ in their physical data arrangement, compression algorithms, null values, and processing precision. This can lead to situations where even if the data content is completely identical, the different storage formats may generate drastically different binary byte streams, resulting in mismatched checksums.

[0099] This application provides a method for verifying low-latency online heterogeneous data in HTAP databases. It eliminates the need for users to modify or migrate existing data table structures, flexibly adapting to user business needs. This overcomes the shortcomings of traditional solutions, such as low efficiency and the need to stop service in practical applications.

[0100] Before introducing the specific methods of this application, the application scenarios of this application will be illustrated by example. It should be understood that the following application scenarios are merely illustrative and are not limited to these examples.

[0101] Scenario 1: The number of data tables in the target database is equal to the number of data tables in the source database;

[0102] The source and target databases have the exact same number of tables, representing a one-to-one full synchronization. This scenario is common in system migrations within the same business line and data mirroring backups, requiring a full mapping of the table structures between the source and target databases after adaptation. The synchronization process needs to address key issues such as data type conversion, field name mapping, and constraint rule adaptation between the databases.

[0103] For example, for ease of understanding, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating a data table synchronization scenario in an embodiment of this application. As shown in the figure, assume that the source database A includes four data tables: a user information table, an order table, a product table, and a logistics table. The target database A also includes four corresponding data tables: a user set table corresponding to the user information table, an order information table corresponding to the order table, a product information table corresponding to the product table, and a logistics set table corresponding to the logistics table. Based on the solution provided in this application, consistency verification can be performed between the data tables in the source database A and the data tables in the target database A.

[0104] Scenario 2: The data tables in the target database originate from multiple source databases;

[0105] The target database contains tables sourced from at least two source databases, representing a multi-source data synchronization model. This scenario is common in data aggregation and analysis, core business data extraction, etc., and requires handling issues such as structural differences and data conflicts between data tables from different sources to ensure that the synchronized data is accurate and conforms to business rules.

[0106] For example, for ease of understanding, please refer to Figure 3 , Figure 3This is another schematic diagram illustrating a data table synchronization scenario in this application. As shown in the figure, assume that source database A includes two data tables: a user information table and an order table. Source database B includes two data tables: a product table and a logistics table. Target database A contains four corresponding data tables: a user set table corresponding to the user information table, an order information table corresponding to the order table, a product information table corresponding to the product table, and a logistics set table corresponding to the logistics table. Based on the solution provided in this application, consistency checks can be performed between the data tables in source database A, the data tables in source database B, and the data tables in target database A.

[0107] Scenario 3: All data tables in the target database originate from a single source database;

[0108] The target database consists of a subset of the tables in the source database, representing a single-source data synchronization model. This scenario is common in business data anonymization and data access control, requiring handling of differences in table structures and data format conversions between databases.

[0109] For example, for ease of understanding, please refer to Figure 4 , Figure 4 This is another schematic diagram illustrating a data table synchronization scenario in this application. As shown in the figure, assume that the source database A includes four data tables: a user information table, an order table, a product table, and a logistics table. The target database A contains two corresponding data tables: a user set table corresponding to the user information table and an order information table corresponding to the order table. Based on the solution provided in this application, consistency verification can be performed on some data tables in the source database A and the data tables in the target database A.

[0110] It should be noted that the above application scenarios are merely examples, and the data verification method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0111] The method provided in this application can be applied to... Figure 5The implementation environment shown includes a source server 501, a target server 502, and an intermediate server 503. The source server 501 and the intermediate server 503, and the intermediate server 503 and the target server 502, can communicate via a network. The network uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, private networks, or any combination of virtual private networks. In some embodiments, customized or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0112] The source server 501 involved in this application includes a source database, which can be an OLTP database, and the source database stores source data tables (e.g., a second data table).

[0113] The target server 502 involved in this application includes a target database, which can be an OLAP database, and the target database stores target data tables (e.g., a first data table).

[0114] The intermediate server 503 involved in this application is deployed with a high-speed link synchronization component (e.g., a CDC component). This high-speed link synchronization component is used to capture and record changes in the database in real time. This real-time capture provides the advantage of not needing to write locks to the source database tables during data synchronization. The high-speed link synchronization component connects the source database and the target database and provides capabilities such as data filtering, type mapping, data consolidation, and heterogeneous data transformation. The CDC component completes hot data synchronization between heterogeneous databases. Specifically, during full data migration, log collection is enabled, and data is pulled from the upstream source database and directly written to the downstream target database. During incremental synchronization, log data from the source database is pulled, and the CDC component parses the log data into statements that the target database can recognize. Finally, the log data is replayed in the target database to complete the data synchronization task.

[0115] In the above implementation environment, in step A1, the CDC component in the intermediate server 503 obtains the basic row count of the first data table in the target database at the first moment. In step A2, the CDC component in the intermediate server 503 obtains the first statistical information of the second data table in the source database from the first moment to the second moment by parsing the log data of the source database. The first statistical information includes the first number of newly added rows and the first number of deleted rows. In step A3, the CDC component in the intermediate server 503 obtains the real-time row count of the first data table at the second moment. In step A4, the CDC component in the intermediate server 503 determines the statistical row count of the first data table at the second moment based on the basic row count, the first number of newly added rows, and the first number of deleted rows at the first moment. In step A5, the CDC component in the intermediate server 503 performs a consistency check between the real-time row count and the statistical row count of the first data table at the second moment, obtaining the first check result of the first data table.

[0116] Based on the above introduction, the data verification method in this application will be described below. Please refer to [link / reference]. Figure 6 The data verification method in this application embodiment can be completed independently by the server, independently by the terminal, or jointly by the terminal and the server. The method of this application includes:

[0117] S601. Obtain the number of basic rows in the first data table of the target database at the first moment;

[0118] In one or more embodiments, the target database typically stores multiple data tables. This application uses the first data table stored in the target database as an example for illustration, but this should not be construed as limiting the application. It should be noted that the target database can be an OLAP database or other databases, such as a data warehouse, data lake, non-relational database (not only SQL, NoSQL) database, OLTP database, etc.

[0119] Specifically, this application uses the CDC component as an example of a high-speed link synchronization component, but it can be extended to other data synchronization components, and is not limited here. The CDC component requests the base row count (base_rows) of the first data table from the target database at the first moment. Here, the first moment is the start time of the verification period. The base row count can be represented as an unsigned 64-bit integer (UInt64) value, and the base row count can be the number of data rows when the full phase is completed, or the number of data rows after being reset when the data row count verification is completed at a certain checkpoint and the data verification results are consistent.

[0120] S602. Obtain the first statistical information of the second data table in the source database from the first time to the second time, wherein the second data table is a heterogeneous data table or a homogeneous data table corresponding to the first data table, the second time occurs after the first time, and the first statistical information includes the first number of newly added rows and the first number of deleted rows obtained from the log data statistics of the source database.

[0121] In one or more embodiments, the CDC component can obtain log data from the source database at each checkpoint and perform statistics on the log data extracted at each checkpoint within a verification cycle to obtain first statistical information of the second data table from the first time to the second time.

[0122] Specifically, the verification period begins at time 1 and ends at time 2. Based on this, the log data obtained at each checkpoint between time 1 and time 2 is filtered to obtain log data for the second data table. Based on this log data, the first number of newly inserted rows (insert_rows) and the first number of deleted rows (delete_rows) in the second data table during the verification period can be calculated. The first number of newly inserted rows can be represented as a UInt64 value, and the first number of deleted rows can also be represented as a UInt64 value.

[0123] It should be noted that the second data table and the first data table describe the same type of data or related data. The table structures of the two tables need to be consistent, but the underlying storage engines can be different (i.e., row storage or column storage).

[0124] S603. Obtain the real-time row count of the first data table at the second moment;

[0125] In one or more embodiments, the CDC component requests the real-time row count (real_rows) of the first data table from the target database at a second time. The real-time row count can be represented as a UInt64 value, and it represents the actual number of data rows in the first data table at the second time.

[0126] S604. Based on the basic number of rows, the first number of newly added rows, and the first number of deleted rows at the first time step, determine the number of statistical rows in the first data table at the second time step.

[0127] In one or more embodiments, the CDC component can determine the statistical row count of the first data table at the second time based on the base row count of the first data table at the first time and the first statistical information of the second data table from the first time to the second time (i.e., the first number of newly added rows and the first number of deleted rows). The statistical row count can be represented as a UInt64 value, and the statistical row count is an estimated number of data rows in the first data table at the second time.

[0128] S605. Perform a consistency check between the real-time row count and the statistical row count of the first data table at the second time and obtain the first check result of the first data table.

[0129] In one or more embodiments, the real-time row count of the first data table at a second time point is compared with the statistical row count at the second time point. If the real-time row count and the statistical row count are the same, the first verification result of the first data table indicates that the first data table has passed the consistency check. If the real-time row count and the statistical row count are different, the first verification result of the first data table indicates that the first data table has failed the consistency check.

[0130] Specifically, for ease of understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the consistency check of data tables in this embodiment of the application. As shown in the figure, after obtaining log data from the source database, the CDC component parses it to obtain the statistical information corresponding to each data table. Simultaneously, the CDC component requests the base row count and real-time row count of each data table from the target database based on Data Manipulation Language (DML). Taking the data table identified as "db1.table1" as an example, the base row count at the first moment and the real-time row count at the second moment are obtained, and the first newly added row count and the first deleted row count of the data table are counted. Based on this, the base row count, the first newly added row count, and the first deleted row count are calculated to obtain the statistical row count. If the statistical row count is inconsistent with the real-time row count, the corresponding error handling mode can be triggered.

[0131] This application provides a data verification method. Using this method, data verification tasks can be performed efficiently and quickly without querying the source database or modifying the source data table structure.

[0132] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, obtaining the basic row count of the first data table in the target database at a first moment may specifically include:

[0133] Send a first query statement to the target database, wherein the first query statement includes the database name of the target database and the table name corresponding to the first data table;

[0134] Receive the first query result returned by the target server, wherein the first query result includes the basic row count of the first data table at the first moment, and the first moment is the moment when the target database responds to the first query statement by performing a row count query on the first data table.

[0135] In one or more embodiments, a method for obtaining the basic row count of a data table based on a target database is described. As can be seen from the foregoing embodiments, the CDC component can obtain the basic row count of the first data table returned by the target database at a first moment by sending a first query statement to the target database.

[0136] Specifically, if the CDC component has obtained query permissions for the target database, it calls the query template, that is:

[0137] SELECT COUNT( from {database_name}.{table_name};

[0138] Among them, SELECT COUNT( `order_db` is a DML query used to count the total number of records in a database table. `{database_name}` represents the database name, for example, `order_db`. `{table_name}` represents the table name, for example, `order_info`.

[0139] Replace `{database name}` in the query template with the database name of the target database, and replace `{table name}` with the table name corresponding to the first data table. This yields the first query statement. Then, connect to the target database server using a database client tool or command-line tool. Execute the first query statement; the target database will then return the first query result, for example, `COUNT(`. = 15823, meaning that the basic number of rows in the first data table at the first moment is 15823. Here, the first moment is the moment when the target database begins to respond to the first query statement and performs a row count query on the first data table, and the first moment is also the start time of the verification period.

[0140] As is understandable, DML is a subset of SQL, primarily used for manipulating data in a database. This includes commands such as INSERT, SELECT, UPDATE, and DELETE to perform CRUD operations on data. SQL is the standard programming language used to manage relational databases, for querying, inserting, updating, and deleting data within the database.

[0141] It should be noted that the term "in response to" in this application refers to the conditions or states upon which the execution of an operation depends, and one or more operations that can be executed when certain conditions or states are met. These operations can be real-time or have a certain delay.

[0142] Secondly, this application provides a method for obtaining the basic row count of a data table based on the target database. By utilizing the native statistical capabilities of the target database, the basic row count can be directly returned based on the first query statement. This method eliminates the need for complex logic configuration, resulting in high execution efficiency and accurate, reliable results. Furthermore, it is applicable to mainstream relational databases without requiring adaptation to different database syntaxes, thus reducing operational complexity and demonstrating strong versatility.

[0143] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, obtaining the real-time row count of the first data table at a second time may specifically include:

[0144] Send a second query statement to the target database, wherein the second query statement includes the database name of the target database and the table name corresponding to the first data table;

[0145] Receive the second query result returned by the target server, wherein the second query result includes the real-time row count of the first data table at the second moment, and the second moment is the moment when the target database responds to the second query statement by querying the row count of the first data table.

[0146] In one or more embodiments, a method for obtaining the real-time row count of a data table based on a target database is described. As can be seen from the foregoing embodiments, the CDC component can obtain the real-time row count of the first data table returned by the target database at a second time by sending a second query statement to the target database.

[0147] Specifically, the CDC component can call the query template again, replacing `{database name}` with the database name of the target database and `{table name}` with the table name corresponding to the first data table, thus obtaining the second query statement. Then, it connects to the target database's server using a database client tool or command-line tool. Finally, it executes the second query statement, and the target database returns the second query result, for example, `COUNT(`. = 11223, meaning that the real-time row count of the first data table at the second moment is 11223. Here, the second moment is the moment when the target database begins responding to the second query statement and performing a row count query on the first data table, and it is also the end time of the verification period.

[0148] Secondly, this application provides a method for obtaining the real-time row count of a data table based on the target database. By utilizing the native statistical capabilities of the target database, the real-time row count can be directly returned based on a second query statement. This method offers high execution efficiency and accurate, reliable results. Furthermore, it is applicable to mainstream relational databases, thereby reducing operational complexity and demonstrating strong versatility.

[0149] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, obtaining the first statistical information of the second data table in the source database from the first time point to the second time point may specifically include:

[0150] For the second data table in the source database, obtain the statistical information corresponding to each checkpoint from the first time to the second time. The checkpoint is used to indicate the time when the log data of the source database is statistically analyzed. The incremental log data is the log data within the check period. The statistical information includes the number of new rows and the number of deleted rows in the second data table within the check period.

[0151] The number of newly added rows in the statistics corresponding to each checkpoint is summed to obtain the first number of newly added rows in the second data table from the first time point to the second time point;

[0152] The number of deleted rows in the statistics corresponding to each checkpoint is summed to obtain the first number of deleted rows in the second data table from the first time point to the second time point.

[0153] In one or more embodiments, a method for obtaining first statistical information based on log data is described. As can be seen from the foregoing embodiments, the CDC component sets up multiple checkpoints to support breakpoint resumption and fault recovery. That is, when the CDC component experiences an unexpected interruption, data synchronization will be restored through checkpoints. To ensure the stability of checkpoints, CDC needs to stop writing when a checkpoint is reached. Therefore, accurate statistical information and the number of stable rows in the target database table at the checkpoint time can be obtained. Furthermore, the first and second times are also the times corresponding to the checkpoints.

[0154] Specifically, the CDC component retrieves log data from the source database at the checkpoint corresponding to the first time point (e.g., checkpoint_1). Then, the CDC component retrieves log data from the source database at the checkpoint corresponding to the next time point (e.g., checkpoint_2). This log data is incremental log data generated from the time point corresponding to checkpoint_1 to the time point corresponding to checkpoint_2. Therefore, the incremental log data can be used to determine the statistics at checkpoint_2, which include at least the number of newly added rows and the number of deleted rows.

[0155] For example, a log record corresponding to a new event can be represented as:

[0156] INSERT INTO {table_name} (field_name1, field_name2, ...) VALUES (value1, value2, ...);

[0157] For example, a log entry corresponding to a deletion event can be represented as:

[0158] DELETE FROM {table name} WHERE qualification;

[0159] For easier understanding, please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating statistical information obtained based on checkpoint aggregation in an embodiment of this application. As shown in the figure, assuming t1 is the first time point and t4 is the second time point, log data can be obtained once at each checkpoint. Please refer to Table 1, which shows... Figure 8 The illustration shows a statistical representation of each checkpoint against the second data table.

[0160] Table 1

[0161]

[0162] As shown in Table 1, since the first moment (i.e., moment t1) is the initial moment of the verification period, the number of newly added rows and the number of deleted rows corresponding to checkpoint 1 (i.e., checkpoint_1) are both 0. Within the verification period from t1 to t2, based on the incremental log data, the number of newly added rows is 20 and the number of deleted rows is 3. Within the verification period from t2 to t3, based on the incremental log data, the number of newly added rows is 358 and the number of deleted rows is 0. Within the verification period from t3 to t4, based on the incremental log data, the number of newly added rows is 0 and the number of deleted rows is 128.

[0163] Therefore, by summing the number of newly added rows corresponding to each checkpoint, we obtain the first number of newly added rows in the second data table from the first time point to the second time point. For example, 0 + 20 + 358 + 0 = 378, meaning the first number of newly added rows is 378. Similarly, by summing the number of deleted rows corresponding to each checkpoint, we obtain the first number of deleted rows in the second data table from the first time point to the second time point. For example, 0 + 3 + 0 + 128 = 131, meaning the first number of deleted rows is 131.

[0164] It should be noted that the log data involved in this application can be a binary log (binlog), where binlog is a type of log file in MySQL databases that records all database modification operations (such as data updates, table structure changes, etc.) in binary format. It is mainly used for master-slave replication and data recovery, and data synchronization and rollback can be achieved by replaying binlog.

[0165] Secondly, this application provides a method for obtaining first statistical information based on log data. By summarizing the statistical information collected at each checkpoint within the verification period, first statistical information reflecting the overall data changes within the verification period is obtained, thereby improving the feasibility of the operation.

[0166] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, the number of statistical rows in the first data table at the second time is determined according to the basic number of rows, the first number of newly added rows, and the first number of deleted rows at the first time. Specifically, this may include:

[0167] The incremental row count is obtained by summing the base row count and the first newly added row count at the first moment.

[0168] The difference between the incremental row count and the first deleted row count is used to obtain the statistical row count of the first data table at the second time step.

[0169] In one or more embodiments, a method for calculating the number of statistical rows is described. As can be seen from the foregoing embodiments, the CDC component can determine the number of statistical rows in the first data table at the second time based on the basic row count of the first data table at the first time and the first statistical information of the second data table from the first time to the second time.

[0170] Specifically, the number of rows in the data table at the end of the verification period can be calculated as follows:

[0171] statistics_row = base_rows + insert_rows – delete_rows; formula (1)

[0172] Here, `statistics_row` represents the number of rows in the target database table at the end of the verification period, for example, the number of rows in the first table at the second time. `base_rows` represents the base number of rows in the target database table at the beginning of the verification period, for example, the number of newly added rows in the second table from the first time to the second time. `insert_rows` represents the number of newly added rows in the source database table during the verification period, for example, the number of newly added rows in the second table from the first time to the second time. `base_rows + insert_rows` represents the incremental row count. `delete_rows` represents the number of deleted rows in the source database table during the verification period, for example, the number of newly deleted rows in the second table from the first time to the second time.

[0173] Secondly, this application provides a method for calculating the number of statistical rows. Using this method, the number of statistical rows in the data table at a given time can be directly calculated using relevant formulas. This eliminates the need for manual row-by-row counting, thus enabling rapid acquisition of the desired results, significantly improving statistical efficiency, and offering strong versatility.

[0174] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, the number of statistical rows in the first data table at the second time is determined according to the basic number of rows, the first number of newly added rows, and the first number of deleted rows at the first time. Specifically, this may include:

[0175] If the number of newly added rows is equal to the number of newly deleted rows, the number of basic rows at the first time step will be used as the number of statistical rows in the first data table at the second time step.

[0176] If the number of newly added rows is not equal to the number of newly deleted rows, the number of basic rows at the first time step is summed with the number of newly added rows to obtain the incremental row count. The difference between the incremental row count and the number of deleted rows is then calculated to obtain the statistical row count of the first data table at the second time step.

[0177] In one or more embodiments, another method for calculating the number of rows to be counted is introduced. As can be seen from the foregoing embodiments, the number of newly added rows and the number of deleted rows in the second data table of the CDC component from the first time point to the second time point can be determined by first determining whether the number of newly added rows and the number of deleted rows are equal, and this can be used as the basis for determining the number of rows to be counted.

[0178] Specifically, if the number of newly added rows is equal to the number of deleted rows, it means that the number of data rows in the second data table has not changed during the verification period. Therefore, the basic number of rows in the first data table at the first moment is directly used as the statistical number of rows in the second moment. If the number of newly added rows is not equal to the number of deleted rows, it means that the number of data rows in the second data table has not changed during the verification period. Therefore, the statistical number of rows in the first data table at the second moment can be calculated using formula (1), which will not be elaborated here.

[0179] Secondly, this application provides another method for calculating the number of rows to be counted. Using the above method, when the number of newly added rows is equal to the number of deleted rows, the base number of rows can be directly used as the number of rows to be counted. This improves computational efficiency and saves computational resources.

[0180] Optionally, in the above Figure 6 In addition to one or more corresponding embodiments, another optional embodiment provided in this application may further include:

[0181] Obtain the base row count of the first data table at the second time step;

[0182] Obtain second statistical information from the second time point to the third time point from the second data table, wherein the third time point occurs after the second time point, and the duration from the second time point to the third time point is equal to the duration from the first time point to the second time point. The second statistical information includes the second number of newly added rows and the second number of deleted rows obtained from the log data statistics of the source database.

[0183] Get the real-time row count of the first data table at the third time point;

[0184] Based on the number of basic rows, the number of newly added rows, and the number of deleted rows at the second time step, determine the number of statistical rows in the first data table at the third time step;

[0185] The consistency of the real-time row count of the first data table at the third time point with the statistical row count at the third time point is checked to obtain the second check result of the first data table.

[0186] In one or more embodiments, a method for performing consistency checks on data tables is described. As described in the foregoing embodiments, within a check cycle, the CDC component collects DML information from log events and organizes the data at the data table level. When the synchronization delay meets the maximum consistency check delay and the time since the last check reaches a predetermined check interval (e.g., the check cycle defaults to once every 24 hours), the CDC component compares the collected number of rows with the actual number of rows stored in real time. If the two are inconsistent, the CDC component will handle the error according to a predetermined pattern.

[0187] Specifically, statistics on the data stream are periodically persisted to system tables in the target-side columnar database along with checkpoints. For example, a statistic may include the following fields:

[0188] The cluster identifier field (cdc_id) is a string type used to uniquely identify the data replication cluster;

[0189] The database name field (db_name) is a string type and represents the database name of the database to which the data table to be checked for consistency belongs.

[0190] The table name field (tb_name) is a string type and represents the name of the table that needs to be checked for consistency.

[0191] The base_rows field, which uses an unsigned 64-bit integer type, is used to record the base number of rows in the data table;

[0192] The `insert_rows` field, using an unsigned 64-bit integer type, is used to count the number of newly added data rows.

[0193] The delete_rows field, an unsigned 64-bit integer, is used to count the number of rows deleted.

[0194] It should be noted that after each data consistency check passes, the base row number field will be reset to the real-time row number of the first data table in the target database at the second moment. At the same time, the stored values ​​of the insert_rows field, delete_rows field, and update_rows field will be cleaned, and the data consistency check for the next check cycle will begin.

[0195] For easier understanding, please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating data consistency verification based on multiple verification cycles in an embodiment of this application. As shown in the figure, assuming t1 is the first time point and t4 is the second time point, and the period from the first time point to the second time point constitutes a complete verification cycle (i.e., verification cycle 1). Based on this, the CDC component calculates the statistical row count of the first data table at time t2 according to the basic row count of the first data table at time t1 and the first statistical information of the second data table within verification cycle 1. Then, the consistency verification between the real-time row count and the statistical row count of the first data table at time t4 is performed to obtain the first verification result of the first data table.

[0196] Next, we proceed to the next verification cycle, namely, verification cycle 2. Verification cycle 2 begins at time 2 and ends at time 3. Assume t6 is time 3, and time t4 to t6 constitute verification cycle 2. Based on this, the CDC component calculates the statistical row count of the first data table at time t6 using the basic row count of the first data table at time t4 and the second statistical information of the second data table within verification cycle 2 (i.e., including the second number of newly added rows and the second number of deleted rows). The calculation method can be found in the aforementioned embodiment and will not be repeated here. Therefore, a consistency check is performed between the real-time row count and the statistical row count of the first data table at time t6 to obtain the second verification result of the first data table.

[0197] It should be noted that, in practical applications, this application also sets a verification timeout duration to control the frequency of data consistency checks. This verification timeout duration is set in seconds to define the verification interval. For example, when the verification timeout duration parameter value is 0, it indicates that no online data consistency check will be performed; when the verification timeout duration parameter value is greater than 0, periodic checks will be performed at the set interval. In this application, the parameter is set to 86400 seconds, and the system performs a data consistency check every 24 hours to ensure the accuracy of data synchronization. Therefore, this design achieves a dynamic balance between performance and data consistency, meeting the needs of different application scenarios.

[0198] Secondly, this application provides a method for performing consistency checks on data tables. Using this method, after obtaining the data check result for one check cycle, the end time of that check cycle is used as the start time of the next check cycle, and data consistency checks continue. This ensures uninterrupted check continuity and guarantees the reliability and consistency of the checks.

[0199] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, obtaining the first statistical information of the second data table in the source database from the first time point to the second time point may specifically include:

[0200] In the event of a fault detection, a second timeframe is determined;

[0201] Obtain the first statistical information of the second data table in the source database from the first time point to the second time point;

[0202] It may also include:

[0203] Under the condition that the fault is recovered, obtain the basic row count of the first data table at the fourth time point, where the fourth time point occurs after the second time point;

[0204] Obtain the third statistical information of the second data table from the fourth time to the fifth time, wherein the fifth time occurs after the fourth time, and the duration from the fourth time to the fifth time is longer than the duration from the first time to the second time. The third statistical information includes the third number of newly added rows and the third number of deleted rows obtained from the log data statistics of the source database.

[0205] Get the real-time row count of the first data table at the fifth time point;

[0206] Based on the number of basic rows, the number of newly added rows, and the number of deleted rows at the fourth time step, determine the number of statistical rows in the first data table at the fifth time step;

[0207] A consistency check is performed between the real-time row count and the statistical row count of the first data table at the fifth time step to obtain the third check result of the first data table.

[0208] In one or more embodiments, a method for performing consistency checks on data tables in the event of a failure is described. As can be seen from the foregoing embodiments, the CDC component may fail during operation. When the CDC component is in a failed state, the target database stops writing data. For example, if the CDC is in a failed state between 10:00 and 11:00, the target database will not write data, but the source database can continue to write data. Based on this, errors may occur when counting the number of newly added and deleted rows during the verification period. The following will describe the situation where inconsistencies occur during consistency checks.

[0209] Scenario 1;

[0210] Specifically, for ease of understanding, please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating a failure during data writing in an embodiment of this application. As shown in the figure, assume that while data table A is being written normally to the source database, its corresponding homogeneous or heterogeneous data table is also being written normally to the target database. At this time, the CDC component sets a checkpoint at time t1 and is about to set another checkpoint at time t2. However, the CDC component at time t... A If a failure occurs between time t1 and time t2, and the system restarts, the log data will be re-counted starting from time t1 (i.e., the log data will be consumed again). However, the log data from time t1 to time t2 will be re-counted. This re-counting of log data should not have been included in the count, leading to statistical errors and causing data inconsistencies that could result in false alarms.

[0211] Scenario 2;

[0212] Specifically, for ease of understanding, please refer to Figure 11 , Figure 11This is another schematic diagram illustrating a failure during the data writing process in an embodiment of this application, such as... Figure 11 As shown in Figure (A), even if it's known which re-statisticalized log data needs to be ignored, data inconsistencies may still occur. Assume that while data table A is being written normally to the source database, its corresponding homogeneous or heterogeneous data table is also being written normally to the target database. At this point, the CDC component restarts due to a failure. After restarting, data can continue being written to both the source and target databases from time t1. However, because checkpoints are set at regular intervals, the timing of the next checkpoint cannot be controlled. For example, the first checkpoint after the restart might be set at time t3.

[0213] The checkpointing mechanism of the CDC component is executed on a time-based basis, but the write speed is jointly determined by the CDC component and the target database. Because the CDC component needs to reload the program after a restart, the initial write speed after restart will be slower than the write speed during normal, stable operation. This leads to a situation where writing data back to the state at the time of the failure restart may take up to t'. A Time. This leads to t' A The timing may be later than time t3, which corresponds to the first checkpoint after the CDC component restarts. Therefore, if a consistency check is performed on data table A and its corresponding homogeneous or heterogeneous data tables at time t3, it will result in inconsistent data verification. For example... Figure 11 As shown in Figure (B), the basic row number at time t1 is Row. A_old The real-time row count at time t4 is Row A_new Therefore, Row A_old Adding the number of inserted rows and subtracting the number of deleted rows does not equal the Row. A_new .

[0214] Therefore, this application addresses the aforementioned problem by resetting the base row count. That is, when a CDC fails, it is necessary to locate log positions after the time of the failure. For example, using... Figure 10For example, assuming the log position corresponding to time t1 is position 1, then the log position after the fault time refers to the log position corresponding to the unexpected restart. After exceeding this log position, the correctness of the number of data rows after the reset point can be guaranteed by resetting the basic row count of the statistics. The way to find this log position is to take the time when the fault occurs as the second time when the fault is detected, and then perform data consistency verification based on the first statistics from the first time to the second time. When the fault is recovered, the latest log position of the source database is taken as the safe position, and the time when the safe position is obtained is taken as the fourth time. Based on this, the basic row count of the first data table at the fourth time and the third statistics of the second data table from the fourth time to the fifth time are obtained. The fourth time to the fifth time is a verification period, and the third statistics include the third number of newly added rows and the third number of deleted rows of the second data table within the verification period. Therefore, based on the basic row count, the third number of newly added rows and the third number of deleted rows at the fourth time, the statistical row count of the first data table at the fifth time can be calculated using formula (1). Finally, the CDC component performs a consistency check between the real-time row count of the first data table at time 5 and the statistical row count at time 5, and obtains the third check result of the first data table.

[0215] Furthermore, for ease of understanding, please refer to Figure 12 , Figure 12 This diagram illustrates the latest log position in the source database. The latest log position (gtid set) can be obtained by executing the `show master status` statement on the source database. When the CDC component restarts after a failure, it resets the statistics and restarts data consistency verification after consuming this log position.

[0216] Secondly, this application embodiment provides a method for reconciling data tables in the event of a failure. Using this method, if the CDC component fails, causing an interruption or restart, the base row count can be directly reset. This ensures that subsequent reconciliations are reliable and accurate.

[0217] Optionally, in the above Figure 6 In addition to one or more corresponding embodiments, another optional embodiment provided in this application may further include:

[0218] Obtain the statistical information corresponding to the checkpoint, including the number of updated rows in the second data table during the check period;

[0219] If the storage conversion conditions are met, the storage space of the second data table is determined based on the number of updated rows in the second data table during each inspection time period. The storage space is either hot storage space or cold storage space.

[0220] In one or more embodiments, a method for adjusting the storage location of a data table based on the number of updated rows is described. As can be seen from the foregoing embodiments, the statistical information obtained based on each checkpoint may also include the number of updated rows. For example, the log record corresponding to an update event can be represented as:

[0221] UPDATE {table_name} SET {field_name} = new_value where(condition);

[0222] For example, another statistic may include the following fields:

[0223] The cluster identifier field (cdc_id) is a string type used to uniquely identify the data replication cluster;

[0224] The database name field (db_name) is a string type and represents the database name of the database to which the data table to be checked for consistency belongs.

[0225] The table name field (tb_name) is a string type and represents the name of the table that needs to be checked for consistency.

[0226] The base_rows field, which uses an unsigned 64-bit integer type, is used to record the base number of rows in the data table;

[0227] The `insert_rows` field, using an unsigned 64-bit integer type, is used to count the number of newly added data rows.

[0228] The `delete_rows` field, using an unsigned 64-bit integer type, is used to count the number of rows deleted.

[0229] The update_rows field, which uses an unsigned 64-bit integer type, is used to count the number of rows updated.

[0230] For example, the conditions for satisfying the storage conversion include at least one of the following:

[0231] In the first case, the total number of updated rows in the second data table is greater than or equal to the row count threshold (e.g., 1 million).

[0232] The second scenario is reaching the preset period (e.g., 7 days, or 30 days, etc.).

[0233] If the storage conversion conditions are met, the update frequency of the second data table can be calculated based on the number of updated rows in each inspection time period, as follows:

[0234] Formula (2)

[0235] Where F represents the update frequency of the data table. This represents the total number of updated rows in the data table across all inspection time periods, i.e. (n represents the total number of time periods to be checked. The value of n can be a preset fixed value or a flexible value.) This indicates the total inspection time span, and a consistent time unit is required. , These represent the duration of the first, second, and nth inspection periods, respectively.

[0236] After calculating the update frequency of the second data table based on formula (2), if the update frequency is higher than or equal to the frequency threshold (e.g., 10 rows / day), the second data table in the source database is stored in hot storage. If the update frequency is lower than the frequency threshold (e.g., 10 rows / day), the second data table in the source database is stored in cold storage.

[0237] It should be noted that the hot storage involved in this application includes, but is not limited to, local disks and solid-state drive (SSD) arrays. Cold storage includes, but is not limited to, network drives, cloud object storage (COS), and hard disk drives (HDDs).

[0238] Secondly, this application provides a method for adjusting the storage location of a data table based on the number of updated rows. Using this method, frequently updated data tables are stored in hot storage, which supports fast read and write of frequently updated data. Infrequently updated data tables are stored in cold storage, which meets the storage needs of infrequently accessed data and avoids inefficiency caused by resource mismatch. Thus, storage efficiency can be adapted, and storage costs optimized.

[0239] Optionally, in the above Figure 6 In addition to one or more corresponding embodiments, another optional embodiment provided in this application may further include:

[0240] Obtain the statistical information corresponding to the checkpoint, including the baseline refresh identifier field of the second data table during the check period;

[0241] If the baseline refresh identifier field included in the statistics indicates that the overlapping phase of full data migration and incremental data migration has ended, the time corresponding to the checkpoint will be used as the first time to perform data consistency verification.

[0242] In one or more embodiments, a method for performing data consistency verification based on a baseline refresh identifier field is introduced. As can be seen from the foregoing embodiments, a challenge in online data verification is ensuring the correctness of the verification itself. CDC components typically involve operations such as resuming interrupted downloads, specifying synchronization objects, and converting full data migration to incremental data migration. These operations often involve replaying some log data. Especially during the process of converting full data migration to incremental data migration, an overlapping phase inevitably occurs; that is, addition, deletion, and modification operations occur between the full start checkpoint and the full end checkpoint. The following will describe the situation where an overlapping phase occurs during the conversion from full data migration to incremental data migration.

[0243] Specifically, for ease of understanding, please refer to Figure 13 , Figure 13 This is a schematic diagram illustrating the process of starting incremental data migration in this embodiment of the application. As shown in the figure, the start time of full data migration corresponds to the full start checkpoint, and the end time of full data migration corresponds to the full end checkpoint. If the incremental data migration phase starts from the full start checkpoint, then the log data between the full start checkpoint and the full end checkpoint will be replayed.

[0244] Please see Figure 14 , Figure 14 This is a schematic diagram illustrating the overlap between full data migration and incremental data migration in an embodiment of this application. As shown in the figure, during the full data migration process, the log data of the source database is constantly changing. The full data migration task captures a static data snapshot at a certain starting checkpoint, while the incremental data migration replays the dynamic log data starting from the full data migration start checkpoint.

[0245] If there's a discrepancy between the full data migration's end checkpoint and the incremental data migration's start checkpoint, DML operations performed within that discrepancy period will be either duplicated or lost. For example, a record might be exported during the full data migration, but this record was updated before the incremental data migration's start point. Therefore, the incremental data migration will replay this update operation. However, if a deletion operation is performed on a record, the incremental data migration might miss the deletion event because the start point is later than the deletion time, leaving a record that should have been deleted in the target database. This data duplication or omission caused by time window discrepancies renders simple row count comparisons meaningless and significantly increases the complexity of accurately locating specific inconsistent data rows.

[0246] Based on this, this application controls the timing of consistency checks on the data table by setting the base refresh identifier field (base_refresh_after_gtid). For example, another statistic may include the following fields:

[0247] The cluster identifier field (cdc_id) is a string type used to uniquely identify the data replication cluster;

[0248] The database name field (db_name) is a string type and represents the database name of the database to which the data table to be checked for consistency belongs.

[0249] The table name field (tb_name) is a string type and represents the name of the table that needs to be checked for consistency.

[0250] The base_rows field, which uses an unsigned 64-bit integer type, is used to record the base number of rows in the data table;

[0251] The `insert_rows` field, using an unsigned 64-bit integer type, is used to count the number of newly added data rows.

[0252] The `delete_rows` field, using an unsigned 64-bit integer type, is used to count the number of rows deleted.

[0253] The update_rows field, which uses an unsigned 64-bit integer type, is used to count the number of rows updated.

[0254] The base refresh flag field (base_refresh_after_gtid), an unsigned 64-bit integer, is used to indicate whether data consistency verification can begin. This field records the log position where the CDC component reconstructed the data statistical baseline. A value of 0 indicates that no baseline refresh operation has been performed; in this case, the data table undergoing data consistency verification has not yet been replayed to the end point of the full data migration log data and therefore cannot participate in data consistency verification. A value greater than 0 indicates that baseline data reconstruction has been completed, meaning it can participate in data consistency verification. The baseline is the log position where incremental data migration begins.

[0255] As can be seen, when the baseline refresh flag field indicates that the overlapping phase of full data migration and incremental data migration has ended, the moment the most recent checkpoint arrives can be taken as the first moment, and data consistency verification can begin based on the first moment. Therefore, during the overlapping phase of full data migration and incremental data migration, no consistency verification will be performed on the data table.

[0256] Secondly, this application provides a method for performing data consistency verification based on a baseline refresh identifier field. Using this method, consistency verification of the data table is not performed during the overlapping phases of full data migration and incremental data migration. This avoids data duplication or omission caused by time window deviations, thereby improving the accuracy and reliability of consistency verification.

[0257] Optionally, in the above Figure 6 In addition to one or more corresponding embodiments, another optional embodiment provided in this application may further include:

[0258] If the first verification result of the first data table indicates that the data verification is consistent, and the second verification result of the first data table indicates that the data verification is inconsistent, obtain the statistical information corresponding to the checkpoint closest to the third time from the second time to the third time. The statistical information also includes the update time of the second data table.

[0259] Based on the second time point and the update time points included in the statistical information, the target time period for which synchronization between the first data table and the second data table fails is determined. The start time of the target time period is the second time point, and the end time of the target time period is the update time points included in the statistical information.

[0260] In one or more embodiments, a method for data consistency verification based on update time is described. As can be seen from the foregoing embodiments, the CDC component can also obtain the update time of the second data table through checkpoints. Here, the update time represents the time when the second data table was last modified.

[0261] For example, another statistic may include the following fields:

[0262] The cluster identifier field (cdc_id) is a string type used to uniquely identify the data replication cluster;

[0263] The database name field (db_name) is a string type and represents the database name of the database to which the data table to be checked for consistency belongs.

[0264] The table name field (tb_name) is a string type and represents the name of the table that needs to be checked for consistency.

[0265] The base_rows field, which uses an unsigned 64-bit integer type, is used to record the base number of rows in the data table;

[0266] The `insert_rows` field, using an unsigned 64-bit integer type, is used to count the number of newly added data rows.

[0267] The `delete_rows` field, using an unsigned 64-bit integer type, is used to count the number of rows deleted.

[0268] The update_rows field, which uses an unsigned 64-bit integer type, is used to count the number of rows updated.

[0269] The base refresh identifier field (base_refresh_after_gtid) is an unsigned 64-bit integer used to mark whether the data has been refreshed after the base point.

[0270] The update time field (update_time), a string type, is used to record the timestamp of the last update of the statistics.

[0271] Based on this, assuming the first verification result indicates data consistency at 13:00:00 on November 18, 2025, and the second verification result indicates data inconsistency at 13:00:00 on November 19, 2025, the CDC component needs to obtain the statistics corresponding to the checkpoint closest to the third time point from the second time point (i.e., 13:00:00 on November 18, 2025) to the third time point (i.e., 13:00:00 on November 19, 2025). For example, if the update time in the statistics corresponding to this checkpoint is 8:00:00 on November 19, 2025, then the target time period for the synchronization failure between the first and second data tables (i.e., data inconsistency) can be determined to be from the second time point (i.e., 13:00:00 on November 18, 2025) to the update time (i.e., 8:00:00 on November 19, 2025).

[0272] Secondly, this application provides a method for data consistency verification based on the update time. Using this method, since the update time represents the last modification of the second data table, it is equivalent to no new, deleted, or updated operations occurring after the update time. Therefore, the scope of data verification can be effectively narrowed.

[0273] Optionally, in the above Figure 6 In addition to one or more corresponding embodiments, another optional embodiment provided in this application may further include:

[0274] Obtain the statistical information corresponding to the checkpoint, including the time when the first data table and the second data table last matched.

[0275] In one or more embodiments, a method for obtaining the time of the most recent checksum agreement is described. As can be seen from the foregoing embodiments, the CDC component can also obtain the time of the most recent checksum agreement between the first data table and the second data table through checkpoints.

[0276] For example, another statistic may include the following fields:

[0277] The cluster identifier field (cdc_id) is a string type used to uniquely identify the data replication cluster;

[0278] The database name field (db_name) is a string type and represents the database name of the database to which the data table to be checked for consistency belongs.

[0279] The table name field (tb_name) is a string type and represents the name of the table that needs to be checked for consistency.

[0280] The base_rows field, which uses an unsigned 64-bit integer type, is used to record the base number of rows in the data table;

[0281] The `insert_rows` field, using an unsigned 64-bit integer type, is used to count the number of newly added data rows.

[0282] The `delete_rows` field, using an unsigned 64-bit integer type, is used to count the number of rows deleted.

[0283] The update_rows field, which uses an unsigned 64-bit integer type, is used to count the number of rows updated.

[0284] The base refresh identifier field (base_refresh_after_gtid) is an unsigned 64-bit integer used to mark whether the data has been refreshed after the base point.

[0285] The update time field (update_time), which is a string type, is used to record the timestamp of the last update of the statistics.

[0286] The checksum_time field, a string type, records the completion time of the last data consistency check.

[0287] Secondly, this application provides a method for obtaining the time of the most recent verification consistency. Using this method, the time range of the verification period can be determined based on the time of the most recent verification consistency, thereby effectively narrowing the scope of data verification.

[0288] Optionally, in the above Figure 6 In addition to one or more corresponding embodiments, another optional embodiment provided in this application may further include:

[0289] Obtain the statistical information corresponding to the checkpoint. The statistical information also includes the log position that is consistent with the most recent verification between the first data table and the second data table. The log position is used to identify the log event in the log data.

[0290] In one or more embodiments, a method for obtaining the most recently verified log position is described. As can be seen from the foregoing embodiments, the CDC component can also obtain the most recently verified log position between the first data table and the second data table through checkpoints. Log positions are used to identify log events in the log data. For example, a log position may be represented as "9c62ca55-7cda-11f0-00f100004023:1742682834", where 9c62ca55-7cda-11f0-00f100004023 represents the identifier of the log data, and 1742682834 represents the 1742682834th event.

[0291] For example, another statistic may include the following fields:

[0292] The cluster identifier field (cdc_id) is a string type used to uniquely identify the data replication cluster;

[0293] The database name field (db_name) is a string type and represents the database name of the database to which the data table to be checked for consistency belongs.

[0294] The table name field (tb_name) is a string type and represents the name of the table that needs to be checked for consistency.

[0295] The base_rows field, which uses an unsigned 64-bit integer type, is used to record the base number of rows in the data table;

[0296] The `insert_rows` field, using an unsigned 64-bit integer type, is used to count the number of newly added data rows.

[0297] The `delete_rows` field, using an unsigned 64-bit integer type, is used to count the number of rows deleted.

[0298] The update_rows field, which uses an unsigned 64-bit integer type, is used to count the number of rows updated.

[0299] The base refresh identifier field (base_refresh_after_gtid) is an unsigned 64-bit integer used to mark whether the data has been refreshed after the base point.

[0300] The update time field (update_time), which is a string type, is used to record the timestamp of the last update of the statistics.

[0301] The checksum_time field, which is a string type, is used to record the completion time of the last data consistency check.

[0302] The checksum_gtid field, a string type, records the log position of the last data consistency check. If the data checksum is inconsistent, it indicates a problem occurred in data processing after that log position.

[0303] Secondly, this application provides a method for obtaining the log position that most recently matched the verification. Using this method, based on the log position that most recently matched the verification, the log data corresponding to the verification period can be determined, thereby effectively narrowing the scope of data verification.

[0304] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, a consistency check is performed on the real-time row count and the statistical row count of the first data table at the second time to obtain a first check result of the first data table, which may specifically include:

[0305] If the consistency verification between the real-time row count and the statistical row count of the first data table at the second moment is not completed within the verification timeout period, the statistical information of the next checkpoint is obtained.

[0306] Based on the basic row count at the first moment and the statistical information at the next checkpoint, determine the number of statistical rows in the first data table at the next checkpoint;

[0307] Get the real-time row count of the first data table at the next checkpoint;

[0308] If the consistency verification between the real-time row count and the statistical row count of the first data table at the next checkpoint is completed within the verification timeout period, the first verification result of the first data table is obtained.

[0309] In one or more embodiments, a method for processing based on a verification timeout duration is described. As can be seen from the foregoing embodiments, in order to meet the requirement of non-intrusiveness to user services, a verification timeout duration that can control data reconciliation is set. The consistency verification will stop if the verification timeout duration is exceeded. The unit of the verification timeout duration is seconds. The default value of the verification timeout duration is 5 seconds, and a verification timeout duration of 0 indicates that no timeout check is performed.

[0310] Specifically, if the consistency verification between the real-time row count and the statistical row count of the first data table at the second time point is not completed within the verification timeout period, the system continues to wait for the next checkpoint. When the next checkpoint arrives, statistical information is generated based on the incremental log data, including the number of newly added rows and the number of deleted rows. Based on this, the CDC component calculates the statistical row count of the first data table according to the base row count at the first time point, the number of newly added rows, and the number of deleted rows corresponding to the adjacent checkpoint, and requests the real-time row count of the first data table at the time corresponding to the adjacent checkpoint. If the consistency verification between the real-time row count and the statistical row count of the first data table at the next checkpoint is completed within the verification timeout period, the first verification result of the first data table is obtained.

[0311] Secondly, this application provides a method for processing based on a verification timeout duration. Using this method, if the consistency verification is not completed within the verification timeout duration, it will be performed again at the next checkpoint. This avoids computational task blocking or memory resource consumption due to network anomalies, thus ensuring smooth system operation and reducing unnecessary waiting and retries. Furthermore, controlling online data verification based on the verification timeout duration ensures completion within a certain timeframe, without affecting user operations.

[0312] Optionally, in the above Figure 6 Based on one or more corresponding embodiments, in another optional embodiment provided by this application, after performing a consistency check on the real-time row count and the statistical row count of the first data table at the second time to obtain the first check result of the first data table, it may further include:

[0313] If the data verification result indicates inconsistency, the operation corresponding to the error mode identifier shall be executed, wherein the operation corresponding to the error mode identifier includes at least one of the following:

[0314] Generate alarm notification messages;

[0315] Interrupt the synchronization between the first and second data tables;

[0316] Stop performing data consistency checks.

[0317] In one or more embodiments, a processing method based on an error mode identifier is described. As can be seen from the foregoing embodiments, to meet the requirement of non-intrusiveness to user services, an error mode identifier is set to control data reconciliation. The error mode identifier is used to indicate the processing method when the data verification results indicate inconsistencies. The following will illustrate this with three verification timeout durations as examples, but this should not be construed as limiting this application.

[0318] (1) The error mode is identified as 0;

[0319] Specifically, when the error mode indicator is 0, it is in a mild alarm mode. In this case, an alarm message is generated, but the synchronization between the first and second data tables is not interrupted (i.e., the synchronization link is not interrupted), and data consistency verification continues.

[0320] (2) The error mode is identified as 1;

[0321] Specifically, when the error mode identifier is 1, it belongs to the severe alarm mode. At this time, an alarm message is generated, and the synchronization between the first data table and the second data table is interrupted (i.e., the synchronization link is interrupted).

[0322] (3) The error mode is identified as 2;

[0323] Specifically, when the error mode is marked as 2, it falls under the reconciliation shutdown mode. In this mode, no alarm message is generated, and the synchronization between the first and second data tables is not interrupted (i.e., the synchronization link is not interrupted). Instead, data consistency verification is stopped. This mode skips the consistency comparison step entirely and does not generate alarm messages, making it suitable for rapid data migration scenarios.

[0324] Secondly, this application provides a method for processing based on error mode identifiers. Through this method, based on pre-configured error mode identifiers, business needs can be flexibly adapted, operational efficiency can be improved, and thus the stable operation of core businesses can be ensured.

[0325] Based on the above embodiments, the overall process will be described below with reference to the illustrations. For easier understanding, please refer to [the relevant documentation / reference]. Figure 15 , Figure 15 The figure shows an overall flowchart of the data verification method in this application embodiment. Specifically:

[0326] In step S1, a checkpoint occurs during the normal incremental data migration process.

[0327] In step S2, it is determined whether the checkpoint has been reconciled (i.e., data consistency verification). If yes, step S3 is executed; otherwise, step S8 is executed.

[0328] In step S3, it is determined whether there has been any change since the last reconciliation. If yes, step S4 is executed; otherwise, step S7 is executed.

[0329] In step S4, it is determined whether the reconciliation time interval is met. If yes, step S5 is executed; otherwise, step S7 is executed.

[0330] In step S5, it is determined whether a reconciliation timeout has occurred. If yes, step S6 is executed; otherwise, step S10 is executed.

[0331] In step S6, continue waiting for the next checkpoint before reconciliation.

[0332] In step S7, if there have been no changes since the last reconciliation, or if the reconciliation time interval is not met, then the data table will not be reconciled.

[0333] In step S8, if reconciliation has not yet been performed, the base number of rows in the target data table is initialized.

[0334] In step S9, the basic row count or reconciliation results are written into the data table of the columnar database system.

[0335] In step S10, the data table is reconciled.

[0336] In step S11, it is determined whether the accounts are consistent. If yes, step S9 is executed; otherwise, step S12 is executed.

[0337] In step S12, the error is handled according to the error handling mode.

[0338] The data verification device in this application is described in detail below. Please refer to [link / reference]. Figure 16 , Figure 16 This is a schematic diagram of one embodiment of the data verification device in this application. The data verification device 160 includes:

[0339] The acquisition module 1601 is used to acquire the number of basic rows of the first data table in the target database at the first moment.

[0340] The acquisition module 1601 is also used to acquire the first statistical information of the second data table in the source database from the first time to the second time, wherein the second data table is a heterogeneous data table or a homogeneous data table corresponding to the first data table, the second time occurs after the first time, and the first statistical information includes the first number of newly added rows and the first number of deleted rows obtained from the log data statistics of the source database.

[0341] The acquisition module 1601 is also used to acquire the real-time row count of the first data table at the second moment;

[0342] The determination module 1602 is used to determine the number of statistical rows in the first data table at the second time based on the basic number of rows, the first number of newly added rows, and the first number of deleted rows at the first time.

[0343] The verification module 1603 is used to verify the consistency between the real-time row count and the statistical row count of the first data table at the second time and obtain the first verification result of the first data table.

[0344] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0345] The acquisition module 1601 is specifically used to send a first query statement to the target database, wherein the first query statement includes the database name of the target database and the table name corresponding to the first data table;

[0346] Receive the first query result returned by the target server, wherein the first query result includes the basic row count of the first data table at the first moment, and the first moment is the moment when the target database responds to the first query statement by performing a row count query on the first data table.

[0347] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0348] The acquisition module 1601 is specifically used to send a second query statement to the target database, wherein the second query statement includes the database name of the target database and the table name corresponding to the first data table;

[0349] Receive the second query result returned by the target server, wherein the second query result includes the real-time row count of the first data table at the second moment, and the second moment is the moment when the target database responds to the second query statement by querying the row count of the first data table.

[0350] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0351] The acquisition module 1601 is specifically used to acquire statistical information corresponding to each checkpoint from the first time to the second time for the second data table in the source database. The checkpoint is used to indicate the time when the log data of the source database is statistically analyzed. The incremental log data is the log data within the check time period. The statistical information includes the number of new rows and the number of deleted rows in the second data table within the check time period.

[0352] The number of newly added rows in the statistics corresponding to each checkpoint is summed to obtain the first number of newly added rows in the second data table from the first time point to the second time point;

[0353] The number of deleted rows in the statistics corresponding to each checkpoint is summed to obtain the first number of deleted rows in the second data table from the first time point to the second time point.

[0354] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0355] Module 1602 is specifically used to sum the base number of rows and the first newly added row number at the first moment to obtain the incremental number of rows;

[0356] The difference between the incremental row count and the first deleted row count is used to obtain the statistical row count of the first data table at the second time step.

[0357] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0358] The determination module 1602 is specifically used to determine the basic row count at the first moment as the statistical row count of the first data table at the second moment when the first number of newly added rows is equal to the first number of deleted rows.

[0359] If the number of newly added rows is not equal to the number of newly deleted rows, the number of basic rows at the first time step is summed with the number of newly added rows to obtain the incremental row count. The difference between the incremental row count and the number of deleted rows is then calculated to obtain the statistical row count of the first data table at the second time step.

[0360] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0361] The acquisition module 1601 is also used to acquire the basic row count of the first data table at the second time step;

[0362] The acquisition module 1601 is also used to acquire second statistical information of the second data table from the second time to the third time, wherein the third time occurs after the second time, and the duration from the second time to the third time is equal to the duration from the first time to the second time. The second statistical information includes the second number of newly added rows and the second number of deleted rows obtained from the log data statistics of the source database.

[0363] The acquisition module 1601 is also used to acquire the real-time row count of the first data table at the third moment;

[0364] The determination module 1602 is also used to determine the number of statistical rows in the first data table at the third time based on the basic number of rows, the second number of newly added rows, and the second number of deleted rows at the second time.

[0365] The verification module 1603 is also used to verify the consistency between the real-time row count of the first data table at the third time and the statistical row count at the third time, and to obtain the second verification result of the first data table.

[0366] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0367] The acquisition module 1601 is specifically used to determine the second moment when a fault is detected;

[0368] Obtain the first statistical information of the second data table in the source database from the first time point to the second time point;

[0369] The acquisition module 1601 is also used to acquire the basic row count of the first data table at the fourth time when the fault is recovered to normal, wherein the fourth time occurs after the second time.

[0370] The acquisition module 1601 is also used to acquire the third statistical information of the second data table from the fourth time to the fifth time, wherein the fifth time occurs after the fourth time, and the duration from the fourth time to the fifth time is longer than the duration from the first time to the second time. The third statistical information includes the third number of newly added rows and the third number of deleted rows obtained from the log data statistics of the source database.

[0371] The acquisition module 1601 is also used to acquire the real-time row count of the first data table at the fifth moment;

[0372] The determination module 1602 is also used to determine the number of statistical rows in the first data table at the fifth time based on the basic row count at the fourth time, the third newly added row count, and the third deleted row count;

[0373] The verification module 1603 is also used to verify the consistency between the real-time row count and the statistical row count of the first data table at the fifth time and obtain the third verification result of the first data table.

[0374] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application, the data verification device 160 further includes a storage module 1604;

[0375] The acquisition module 1601 is also used to acquire the statistical information corresponding to the checkpoint, wherein the statistical information also includes the number of updated rows in the second data table during the inspection period;

[0376] Storage module 1604 is used to determine the storage space of the second data table based on the number of updated rows of the second data table in each inspection time period, provided that the storage conversion conditions are met. The storage space is either hot storage space or cold storage space.

[0377] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0378] The acquisition module 1601 is also used to acquire statistical information corresponding to the checkpoint, wherein the statistical information also includes the baseline refresh identifier field of the second data table during the inspection period.

[0379] The determination module 1602 is also used to determine the time corresponding to the checkpoint as the first time for performing data consistency verification when the baseline refresh identifier field included in the statistical information indicates that the overlapping phase of full data migration and incremental data migration has ended.

[0380] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0381] The acquisition module 1601 is further configured to acquire statistical information corresponding to the checkpoint closest to the third time from the second time to the third time when the first verification result of the first data table indicates that the data verification is consistent and the second verification result of the first data table indicates that the data verification is inconsistent. The statistical information also includes the update time of the second data table.

[0382] The determination module 1602 is further configured to determine the target time period in which the synchronization between the first data table and the second data table fails, based on the second time period and the update time included in the statistical information, wherein the start time of the target time period is the second time period and the end time of the target time period is the update time included in the statistical information.

[0383] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0384] The acquisition module 1601 is also used to acquire the statistical information corresponding to the checkpoint, including the time when the first data table and the second data table were last verified to be consistent.

[0385] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0386] The acquisition module 1601 is also used to acquire the statistical information corresponding to the checkpoint. The statistical information includes the log position that is consistent with the most recent verification between the first data table and the second data table. The log position is used to determine the log event in the log data.

[0387] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application,

[0388] The verification module 1603 is specifically used to obtain the statistical information of the next checkpoint if the consistency verification between the real-time row count and the statistical row count of the first data table at the second time is not completed within the verification timeout period.

[0389] Based on the basic row count at the first moment and the statistical information at the next checkpoint, determine the number of statistical rows in the first data table at the next checkpoint;

[0390] Get the real-time row count of the first data table at the next checkpoint;

[0391] If the consistency verification between the real-time row count and the statistical row count of the first data table at the next checkpoint is completed within the verification timeout period, the first verification result of the first data table is obtained.

[0392] Optionally, in the above Figure 16 Based on the corresponding embodiments, in another embodiment of the data verification device 160 provided in this application, the data verification device 160 further includes an execution module 1605;

[0393] The execution module 1605 is used to perform a consistency check on the real-time row count and the statistical row count of the first data table at the second time. After obtaining the first check result of the first data table, if the data check result indicates that the data is inconsistent, the module executes the operation corresponding to the error mode identifier. The operation corresponding to the error mode identifier includes at least one of the following:

[0394] Generate alarm notification messages;

[0395] Interrupt the synchronization between the first and second data tables;

[0396] Stop performing data consistency checks.

[0397] Figure 17This is a schematic diagram of a computer device structure provided in an embodiment of this application. The computer device 1700 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1722 (e.g., one or more processors) and a memory 1732, and one or more storage media 1730 (e.g., one or more mass storage devices) for storing application programs 1742 or data 1744. The memory 1732 and storage media 1730 can be temporary or persistent storage. The program stored in the storage media 1730 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device. Furthermore, the CPU 1722 may be configured to communicate with the storage media 1730 and execute the series of instruction operations in the storage media 1730 on the computer device 1700.

[0398] Computer device 1700 may also include one or more power supplies 1726, one or more wired or wireless network interfaces 1750, one or more input / output interfaces 1758, and / or one or more operating systems 1741, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0399] The steps performed by the computer device in the above embodiments can be based on this Figure 17 The computer device structure shown.

[0400] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0401] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0402] It is understood that, in the specific embodiments of this application, user permission or consent is required when data such as data tables is used in specific products or technologies. That is, before collecting user data, users can be notified through prompts, pop-ups, or voice prompts. The process of collecting user data only begins after user permission or consent has been obtained. In other words, all user data collected in this application is collected with the user's consent, and the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0403] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0404] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0405] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0406] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0407] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0408] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a server or terminal device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0409] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for data verification, characterized in that, include: Obtain the number of basic rows in the first data table of the target database at the first moment; Obtain first statistical information of the second data table in the source database from the first time point to the second time point, wherein the second data table is a heterogeneous data table or a homogeneous data table corresponding to the first data table, the second time point occurs after the first time point, and the first statistical information includes the first number of newly added rows and the first number of deleted rows obtained based on the log data statistics of the source database. Obtain the real-time row count of the first data table at the second time point; Based on the basic row count, the first newly added row count, and the first deleted row count at the first time point, determine the statistical row count of the first data table at the second time point; A consistency check is performed between the real-time row count of the first data table at the second time and the statistical row count at the second time to obtain the first check result of the first data table.

2. The method according to claim 1, characterized in that, The step of obtaining the basic row count of the first data table in the target database at the first moment includes: Send a first query statement to the target database, wherein the first query statement includes the database name of the target database and the table name corresponding to the first data table; The system receives a first query result returned by the target database, wherein the first query result includes the basic row count of the first data table at the first time, and the first time is the moment when the target database responds to the first query statement by performing a row count query on the first data table.

3. The method according to claim 1, characterized in that, The step of obtaining the real-time row count of the first data table at the second time includes: Send a second query statement to the target database, wherein the second query statement includes the database name of the target database and the table name corresponding to the first data table; The system receives a second query result returned by the target database, wherein the second query result includes the real-time row count of the first data table at the second time, and the second time is the moment when the target database responds to the second query statement by querying the row count of the first data table.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the first statistical information of the second data table in the source database from the first time point to the second time point includes: For the second data table in the source database, obtain the statistical information corresponding to each checkpoint from the first time to the second time. The checkpoint is used to indicate the time when the log data of the source database is statistically analyzed. The incremental log data is the log data within the check time period. The statistical information includes the number of new rows and the number of deleted rows in the second data table within the check time period. The number of newly added rows in the statistical information corresponding to each checkpoint is summed to obtain the first number of newly added rows in the second data table from the first time point to the second time point; The number of deleted rows in the statistics corresponding to each checkpoint is summed to obtain the first number of deleted rows in the second data table from the first time point to the second time point.

5. The method according to any one of claims 1 to 4, characterized in that, Determining the number of rows in the first data table at the second time step based on the base row count, the first newly added row count, and the first deleted row count at the first time step includes: The incremental number of rows is obtained by summing the base number of rows at the first time step and the first newly added row number. The difference between the incremental row count and the first deleted row count is used to obtain the statistical row count of the first data table at the second time point.

6. The method according to any one of claims 1 to 4, characterized in that, Determining the number of rows in the first data table at the second time step based on the base row count, the first newly added row count, and the first deleted row count at the first time step includes: If the number of newly added rows is equal to the number of deleted rows, the number of basic rows at the first moment is taken as the number of statistical rows in the first data table at the second moment; If the number of newly added rows is not equal to the number of newly deleted rows, the number of basic rows at the first time point is summed with the number of newly added rows to obtain the number of incremental rows, and the difference between the number of incremental rows and the number of newly deleted rows is calculated to obtain the number of statistical rows in the first data table at the second time point.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain the base row number of the first data table at the second time step; Obtain second statistical information from the second time point to the third time point from the second data table, wherein the third time point occurs after the second time point, and the duration from the second time point to the third time point is equal to the duration from the first time point to the second time point. The second statistical information includes the second number of newly added rows and the second number of deleted rows obtained based on the log data of the source database. Obtain the real-time row count of the first data table at the third time. Based on the base row count, the second newly added row count, and the second deleted row count at the second time point, determine the statistical row count of the first data table at the third time point; The consistency of the real-time row count and the statistical row count of the first data table at the third time is verified to obtain the second verification result of the first data table.

8. The method according to any one of claims 1 to 6, characterized in that, The step of obtaining the first statistical information of the second data table in the source database from the first time point to the second time point includes: In the event of a fault detection, the second moment is determined; Obtain the first statistical information of the second data table in the source database from the first time point to the second time point; The method further includes: When the fault is recovered, the number of basic rows in the first data table at the fourth time point is obtained, wherein the fourth time point occurs after the second time point; Obtain third statistical information from the fourth time to the fifth time from the second data table, wherein the fifth time occurs after the fourth time, and the duration from the fourth time to the fifth time is greater than the duration from the first time to the second time. The third statistical information includes the third number of newly added rows and the third number of deleted rows obtained based on the log data statistics of the source database. Obtain the real-time row count of the first data table at the fifth time point; Based on the base row count at the fourth time step, the third newly added row count, and the third deleted row count, determine the statistical row count of the first data table at the fifth time step; A consistency check is performed on the real-time row count and the statistical row count of the first data table at the fifth time to obtain the third check result of the first data table.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Obtain the statistical information corresponding to the checkpoint, wherein the statistical information also includes the number of updated rows in the second data table during the check period; If the storage conversion conditions are met, the storage space of the second data table is determined based on the number of updated rows in the second data table during each inspection time period, wherein the storage space is either hot storage space or cold storage space.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Obtain the statistical information corresponding to the checkpoint, wherein the statistical information also includes the baseline refresh identifier field of the second data table within the check time period; If the baseline refresh identifier field included in the statistical information indicates that the overlapping phase of full data migration and incremental data migration has ended, the time corresponding to the checkpoint shall be taken as the first time for performing data consistency verification.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: If the first verification result of the first data table indicates that the data verification is consistent, and the second verification result of the first data table indicates that the data verification is inconsistent, obtain the statistical information corresponding to the checkpoint closest to the third time from the second time to the third time, wherein the statistical information also includes the update time of the second data table; Based on the second time point and the update time point included in the statistical information, a target time period for which synchronization between the first data table and the second data table fails is determined, wherein the start time of the target time period is the second time point, and the end time of the target time period is the update time point included in the statistical information.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: Obtain the statistical information corresponding to the checkpoint, wherein the statistical information also includes the time when the first data table and the second data table were last verified to be consistent.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Obtain the statistical information corresponding to the checkpoint, wherein the statistical information also includes the log position that is consistent with the most recent verification of the first data table and the second data table, and the log position is used to determine the log event in the log data.

14. The method according to any one of claims 1 to 13, characterized in that, The step of verifying the consistency between the real-time row count and the statistical row count of the first data table at the second time and obtaining the first verification result of the first data table includes: If the consistency verification between the real-time row count and the statistical row count of the first data table at the second time point is not completed within the verification timeout period, the statistical information of the next checkpoint is obtained. Based on the basic row count at the first moment and the statistical information at the next checkpoint, determine the number of statistical rows in the first data table at the next checkpoint; Obtain the real-time row count of the first data table at the next checkpoint; If the consistency verification of the real-time row count and the statistical row count of the first data table at the next checkpoint is completed within the verification timeout period, then the first verification result of the first data table is obtained.

15. The method according to any one of claims 1 to 14, characterized in that, After performing a consistency check on the real-time row count and the statistical row count of the first data table at the second time to obtain a first check result for the first data table, the method further includes: If the data verification result indicates data inconsistency, the operation corresponding to the error mode identifier is executed, wherein the operation corresponding to the error mode identifier includes at least one of the following: Generate alarm notification messages; Interrupt the synchronization between the first data table and the second data table; Stop performing data consistency checks.

16. A data verification device, characterized in that, include: The acquisition module is used to obtain the basic row count of the first data table in the target database at the first moment. The acquisition module is further configured to acquire first statistical information of the second data table in the source database from the first time to the second time, wherein the second data table is a heterogeneous data table or a homogeneous data table corresponding to the first data table, the second time occurs after the first time, and the first statistical information includes the first number of newly added rows and the first number of deleted rows obtained based on the log data statistics of the source database. The acquisition module is further configured to acquire the real-time row count of the first data table at the second time. The determination module is used to determine the number of statistical rows in the first data table at the second time based on the basic number of rows at the first time, the first number of newly added rows, and the first number of deleted rows. The verification module is used to verify the consistency between the real-time row count of the first data table at the second time and the statistical row count at the second time, and to obtain the first verification result of the first data table.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the data verification method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data verification method according to any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the data verification method according to any one of claims 1 to 15.

Images