Consistency checking method, device and equipment of heterogeneous database and storage medium

Abstract

The application is suitable for the technical field of database, and relates to a consistency checking method and device of a heterogeneous database, equipment and a storage medium. The transaction logs of a source database and a target database are respectively analyzed in a read-only mode to obtain a first data change event of the source database and a second data change event of the target database; the data in the first data change event and the second data change event is respectively subjected to semantic normalization processing according to a same specification strategy to obtain a first standard data change event and a second standard data change event; the change description data in the first standard data change event and the second standard data change event is respectively allocated a virtual row identifier according to a same allocation strategy; and the first standard data change event and the second standard data change event are compared based on the virtual row identifier to obtain a comparison result. The consistency checking with high precision and low cost can be realized in a complex heterogeneous environment, and the real-time requirement of the consistency checking is met.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a consistency verification method for heterogeneous databases, a consistency verification device for heterogeneous databases, a computer device, and a computer-readable storage medium. Background Technology

[0002] In scenarios involving data migration, synchronization, backup, and disaster recovery, data consistency verification between heterogeneous databases is a critical step in ensuring business continuity. For example, when migrating data records from a source database to a target database, it is necessary to verify whether the data on both ends is consistent after the migration. Related technologies typically employ methods such as dual writes based on business programs or periodic full table scans for data verification. However, these verification methods are no longer sufficient to meet the real-time requirements under high concurrency and large data volumes. Summary of the Invention

[0003] This application provides a method, apparatus, computer device, and computer-readable storage medium for consistency verification of heterogeneous databases, which can achieve high-precision, low-overhead consistency verification in complex heterogeneous environments, thereby meeting the real-time requirements of consistency verification.

[0004] In a first aspect, embodiments of this application provide a method for consistency verification of heterogeneous databases, including: The transaction logs of the source database and the target database are parsed in read-only mode to obtain the first data change event of the source database and the second data change event of the target database. Following the same standard strategy, semantic normalization is performed on the data in the first data change event and the second data change event respectively to obtain the first standard data change event and the second standard data change event; According to the same allocation strategy, virtual row identifiers are assigned to the change description data in the first standard data change event and the second standard data change event, respectively; Based on the virtual row identifier, the first standard data change event and the second standard data change event are compared to obtain the comparison result.

[0005] Secondly, embodiments of this application provide a consistency verification device for heterogeneous databases, comprising: The event acquisition module is used to parse the transaction logs of the source database and the target database in read-only mode to obtain the first data change event of the source database and the second data change event of the target database. The standardization processing module is used to perform semantic standardization processing on the data in the first data change event and the second data change event according to the same standardization strategy, so as to obtain the first standard data change event and the second standard data change event. The identifier allocation module is used to assign virtual row identifiers to the change description data in the first standard data change event and the second standard data change event according to the same allocation strategy. The data verification module is used to compare the first standard data change event and the second standard data change event based on the virtual row identifier to obtain the comparison result.

[0006] Optionally, in one embodiment, the data verification module is used to perform an event-by-event comparison of the first standard data change event and the second standard data change event based on the virtual row identifier when the first event order of the first standard data change event from the source database and the second event order of the second standard data change event from the target database are consistent, and obtain a comparison result; or, when the first event order and the second event order are inconsistent, the first standard data change event and the second standard data change event are organized into micro-batches based on the virtual row identifier to obtain a source micro-batch and a target micro-batch; calculate the first batch summary of the source micro-batch and the second batch summary of the target micro-batch; compare the first batch summary and the second batch summary to obtain a comparison result.

[0007] Optionally, in one embodiment, the data verification module is used to obtain the event delay between the first standard data change event and the second standard data change event corresponding to the same virtual row identifier when the first event sequence of the first standard data change event from the source database and the second event sequence of the second standard data change event from the target database are consistent; when the event delay is less than or equal to a preset delay threshold, the first standard data change event and the second standard data change event are compared event by event based on the virtual row identifier to obtain the comparison result.

[0008] Optionally, in one embodiment, the standardization processing module is used to map the data types in the first data change event and the second data change event to a unified standard data type; and to convert the data values ​​in the first data change event and the second data change event into unified standard data values.

[0009] Optionally, in one embodiment, the change description data includes the primary key value of the data record that has been changed. According to the same allocation strategy, the identifier allocation module is used to perform a hash operation on the primary key value contained in the change description data in the first standard data change event to generate a virtual row identifier for the change description data in the first standard data change event; and to perform the same hash operation on the primary key value contained in the change description data in the second standard data change event to generate a virtual row identifier for the change description data in the second standard data change event.

[0010] Optionally, in one embodiment, the heterogeneous database consistency verification device provided in this application further includes a difference elimination module, which is used to generate and output executable database operation statements for eliminating data differences based on the data differences between the target database and the source database when the comparison result indicates that the target database and the source database are inconsistent.

[0011] Optionally, in one embodiment, the difference exclusion module is used to generate an update operation statement to update the data value of the data record corresponding to the same primary key value in the target database to the data value of the data record corresponding to the same primary key value in the source database when the data difference is that the data value corresponding to the same primary key value in the target database is different; when the data difference is that the data record is missing in the target database, it generates an insert operation statement to insert the corresponding data record in the source database into the target database; if the data difference is that there is redundant data record in the target database, it generates a delete operation statement to delete the redundant data record from the target database.

[0012] Thirdly, the computer device provided in the embodiments of this application includes a processor and a memory. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it implements the heterogeneous database consistency verification method provided in the embodiments of this application.

[0013] Fourthly, the computer-readable storage medium provided in the embodiments of this application stores a computer program, which, when executed by a processor, implements the heterogeneous database consistency verification method provided in the embodiments of this application.

[0014] The data verification scheme provided in this application parses the transaction logs of the source and target databases in read-only mode to obtain a first data change event in the source database and a second data change event in the target database. Following the same standardization strategy, semantic normalization is performed on the data in the first and second data change events to obtain a first standard data change event and a second standard data change event. Virtual row identifiers are assigned to the change description data in the first and second standard data change events according to the same allocation strategy. Based on the virtual row identifiers, the first and second standard data change events are compared to obtain the comparison result. In this way, by performing non-intrusive data collection from the source and target databases, the impact of traditional full-scale comparisons on business performance is avoided. Simultaneously, the combination of semantic normalization and the comparison mechanism based on virtual row identifiers can shield against interference from differences between heterogeneous databases caused by non-business factors such as sorting rules and data type definitions. Therefore, high-precision, low-overhead consistency verification can be achieved in complex heterogeneous environments, meeting the real-time requirements of consistency verification. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of an application environment for the heterogeneous database consistency verification method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the consistency verification method for heterogeneous databases provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of the heterogeneous database consistency verification device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0018] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0019] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0020] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0021] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0023] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0024] To illustrate the technical solution of this application, specific embodiments are described below.

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In related technologies, consistency verification is usually performed using methods such as triggers and timed snapshots to detect the data consistency between the source database and the target database.

[0027] Among them, the trigger-based validation method creates AFTERINSERT / UPDATE / ... on the source database table. A DELETE trigger writes changed data to a "change log table." The validation program polls this table and compares it with the target database. However, this validation method is highly intrusive and has a significant impact on the performance of high-concurrency OLTP systems.

[0028] The snapshot-based verification method periodically (e.g., every 15 minutes) extracts the same batch of data (or the entire dataset) from the source and target databases and compares them. However, this verification method has high latency and cannot detect inconsistencies within minutes; it is moderately invasive, generating a large number of read-only queries during data extraction, consuming IO and CPU, and affecting peak business performance; and it has limited throughput, with full or large-scale sampling taking a very long time.

[0029] In view of the shortcomings of the prior art, the heterogeneous database consistency verification method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the computer device communicates with the source database and the target database via a network. The computer device can parse the transaction logs of the source database and the target database in read-only mode to obtain a first data change event in the source database and a second data change event in the target database. Following the same standardization strategy, semantic normalization processing is performed on the data in the first and second data change events to obtain a first standard data change event and a second standard data change event. Following the same allocation strategy, virtual row identifiers are assigned to the change description data in the first and second standard data change events respectively. Based on the virtual row identifiers, the first and second standard data change events are compared to obtain the comparison result. The following detailed description of this application is provided through specific embodiments.

[0030] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a heterogeneous database consistency verification method provided in an embodiment of this application, as shown below. Figure 2 As shown, the flow of this heterogeneous database consistency verification method can be as follows: In S110, the transaction logs of the source database and the target database are parsed in read-only mode to obtain the first data change event of the source database and the second data change event of the target database.

[0031] In data migration, synchronization, backup, and disaster recovery scenarios, data records are transferred between at least two databases. The source database is the database from which the data transfer begins, while the target database is the database from which it ends. For example, in a data migration scenario, the source database is the original database to be migrated, and the target database is the new database that will support the business operations after the migration. In a data synchronization scenario, the source database is the upstream master database, and the target database is the downstream slave database or a remote replica database. In a backup scenario, the source database is the production database, and the target database is the backup database. In a disaster recovery scenario, the source database is the primary database, and the target database is the disaster recovery database. For instance, in a data migration scenario, if a company needs to migrate a user's data table from an Oracle database to a heterogeneous YashanDB database, then the Oracle database is the source database, and the YashanDB database is the target database.

[0032] It should be noted that the division between the source database and the target database is not absolutely fixed, but is dynamically determined according to the specific business process and data flow. The same database may serve as both a source database and a target database in different scenarios. For example, in a bidirectional synchronization architecture, when database A synchronizes with database B, A is the source database and B is the target database; while when database B synchronizes with database A, database B becomes the source database and database A becomes the target database.

[0033] The following embodiments use a computer device as the execution subject to describe in detail the consistency verification method for heterogeneous databases provided in this application.

[0034] In this embodiment, independent Change Data Capture (CDC) components can be pre-deployed on the source and target databases. These components continuously monitor and parse the transaction logs (such as Oracle's REDO logs, MySQL's BINLOG logs, and PostgreSQL's WAL logs) of their respective databases in read-only mode. When a data change operation occurs, the CDC component can capture the operation and generate a data change event describing it. This event includes, but is not limited to, the transaction identifier, logical row identifier, operation type (such as update, insert, delete), and before-and-after image data of the affected data records (i.e., the image data before and after the change).

[0035] Accordingly, the computer device will receive data change events captured by the change data capture agent component deployed in the source database, denoted as the first data change event, and will receive data change events captured by the change data capture agent component deployed in the target database, denoted as the second data change event.

[0036] It should be noted that due to factors such as network transmission delay and uncertainty of transaction commit order, the arrival time of the first data change event and the second data change event corresponding to the same data change operation may differ. It is possible that the first data change event arrives before the second data change event, or the second data change event arrives before the first data change event, or the first data change event and the second data change event arrive at the same time.

[0037] In S120, the data in the first data change event and the second data change event are semantically normalized according to the same standard strategy to obtain the first standard data change event and the second standard data change event.

[0038] It should be noted that because the source and target databases may use different storage engines, character sets, or data type definitions, the captured data change events may differ in format, encoding, or numerical representation. For example, floating-point numbers in the source database may be stored as fixed-point numbers in the target database. This heterogeneity at the underlying storage level means that directly comparing the original data change events often fails to accurately reflect the consistency of business logic.

[0039] Therefore, in this embodiment, after obtaining a first data change event from the source database, the computer device performs semantic normalization processing on the data in the first data change event according to the configured normalization strategy to obtain a first standard data change event. Similarly, after obtaining a second data change event from the target database, the computer device performs semantic normalization processing on the data in the second data change event according to the same normalization strategy to obtain a second standard data change event, thereby ensuring that the first standard data change event and the second standard data change event are semantically comparable. The specific implementation of the "normalization strategy" is not limited here, as long as it can eliminate semantic ambiguity caused by heterogeneous storage.

[0040] In S130, virtual row identifiers are assigned to the change description data in the first standard data change event and the second standard data change event according to the same allocation strategy.

[0041] As described above, after the computer device completes the semantic normalization processing of the first data change event and obtains the corresponding first standard data change event, it assigns a virtual row identifier to the first standard data change event according to the configured allocation strategy. This virtual row identifier uniquely identifies the corresponding data change operation. Similarly, after the computer device completes the semantic normalization processing of the second data change event and obtains the corresponding second standard data change event, it assigns a virtual row identifier to the second standard data change event according to the same allocation strategy. This virtual row identifier uniquely identifies the corresponding data change operation. In this way, a logical mapping relationship can be established between the first and second standard data change events through the virtual row identifier, enabling accurate matching of data changes originating from the same business operation based on the virtual row identifier, thereby achieving accurate comparison across heterogeneous databases.

[0042] In S140, the first standard data change event and the second standard data change event are compared based on the virtual row identifier to obtain the comparison result.

[0043] As described above, after the virtual row identifier is assigned, the computer device uses the virtual row identifier as the association key to match the first standard data change event with the second standard data change event, identify the data change pairs originating from the same business operation, and perform content comparison on the successfully matched data change pairs to obtain the corresponding comparison results. These comparison results are used to characterize the consistency status of the source database and the target database at the business semantic level.

[0044] As described above, the data scheme provided in this application parses the transaction logs of the source and target databases in read-only mode to obtain the first data change event of the source database and the second data change event of the target database. Following the same standardization strategy, semantic normalization is performed on the data in the first and second data change events to obtain the first standard data change event and the second standard data change event. Following the same allocation strategy, virtual row identifiers are assigned to the change description data in the first and second standard data change events. Based on the virtual row identifiers, the first and second standard data change events are compared to obtain the comparison results. Thus, by performing non-intrusive data collection from the source and target databases, the impact of traditional full-scale comparisons on business performance is avoided. Simultaneously, the combination of semantic normalization and the comparison mechanism based on virtual row identifiers can shield against differences caused by non-business factors such as sorting rules and data type definitions between heterogeneous databases. This enables high-precision, low-overhead consistency verification in complex heterogeneous environments, meeting the real-time requirements of consistency verification.

[0045] Optionally, in one embodiment, semantic normalization processing is performed on the data in the first data change event and the second data change event respectively to obtain the first standard data change event and the second standard data change event, including: Map the data types in the first data change event and the second data change event to a unified standard data type; The data values ​​in the first data change event and the second data change event are converted into unified standard data values ​​to obtain the first standard data change event and the second standard data change event.

[0046] Based on the above embodiments, this application further provides an optional specification strategy, including data type mapping and data value conversion, to eliminate structural differences between heterogeneous databases.

[0047] Specifically, the computer equipment maps the data types in the first and second data change events to a unified standard data type. This standard data type is a predefined intermediate data type system covering all common database types, including but not limited to CANONICAL_INT, CANONICAL_STRING, CANONICAL_DECIMAL, and CANONICAL_TIMESTAMP. Each standard data type contains complete metadata descriptions, such as precision, scale, character set, and time zone information. For example, the DATETIME type in the source database and the DATE type in the target database can be mapped to the standard timestamp type CANONICAL_TIMESTAMP to eliminate time zone and format differences; the VARCHAR type in the source database and the STRING type in the target database can be mapped to the standard string type CANONICAL_STRING to eliminate character set encoding differences; the FLOAT type in the source database and the DOUBLE PRECISION type in the target database can be mapped to the standard floating-point type CANONICAL_DECIMAL to eliminate precision representation differences, and so on.

[0048] On the other hand, the computer equipment also converts the data values ​​in the first and second data change events into unified standard data values. The conversion of standard data values ​​follows predefined conversion rules. For example, timestamps can be unified into UTC format strings, numerical values ​​can be unified into decimal strings with specified precision and scale, strings can be unified into a standardized form of UTF-8 encoding, and so on.

[0049] Through the above dual normalization processing based on data type and data value, the heterogeneity of the underlying database engine in terms of data type definition, storage format and precision representation can be effectively shielded, providing a unified and comparable data foundation for subsequent data change event comparison, improving the consistency and accuracy of data comparison, and thus ensuring the integrity and reliability of the data synchronization process.

[0050] Optionally, in one embodiment, based on the virtual row identifier, a comparison is made between the first standard data change event and the second standard data change event to obtain a comparison result, including: If the first event sequence of the first standard data change event from the source database is consistent with the second event sequence of the second standard data change event from the target database, then the first standard data change event and the second standard data change event are compared event by event based on the virtual row identifier to obtain the comparison result; If the order of the first event and the order of the second event are inconsistent, the first standard data change event and the second standard data change event are organized into micro-batches based on the virtual row identifier to obtain the source micro-batch and the target micro-batch; the first batch summary of the source micro-batch and the second batch summary of the target micro-batch are calculated; the first batch summary and the second batch summary are compared to obtain the comparison result.

[0051] Based on the above embodiments, this application provides an optional differentiated event comparison strategy.

[0052] In this embodiment of the application, in order to improve the comparison efficiency, after obtaining the first standard data change event and the second standard data change event, the hash values ​​of the normalized before and after image data in the first standard data change event and the second standard data change event are calculated according to the same hash algorithm, and then appended to the corresponding standard data change event to form an event summary.

[0053] Accordingly, during the comparison phase, the computer equipment first identifies whether the first event sequence of the first standard data change events from the source database and the second event sequence of the second standard data change events from the target database are consistent. The event sequence can be determined by the order of the virtual row identifiers of their respective standard data change events. It should be noted that consistency between the first and second event sequences does not mean they are completely identical; they only need to maintain a consistent overall trend. For example, if more than 80% of the events are in the same sequence, they can be considered consistent.

[0054] Specifically, when the order of the first and second events is consistent, it indicates that the data change events in the source and target databases are synchronized in time. In this case, a step-by-step comparison can be performed directly based on the virtual row identifier. That is, following the same event order, the event digests carried by the first standard data change event with the same virtual row identifier are compared with the event digests carried by the second standard data change event. If the event digests are consistent, the data record in the source database corresponding to the first standard data change event is consistent with the data record in the target database corresponding to the second standard data change event. If the event digests are inconsistent, it indicates that there is a difference between the data record in the source database corresponding to the first standard data change event and the data record in the target database corresponding to the second standard data change event.

[0055] Furthermore, when the order of the first event is inconsistent with the order of the second event, it indicates a temporal discrepancy between the data change events in the source and target databases. If event-by-event comparison is still used in this case, misjudgment is likely due to the out-of-order sequence. Therefore, the computer device organizes the first and second standard data change events into micro-batches based on virtual row identifiers, resulting in source-side and target-side micro-batches. The source-side micro-batchens contain multiple first standard data change events, ordered according to the virtual row identifier; the target-side micro-batchens contain the same number of second standard data change events as the source-side micro-batchens, also ordered according to the virtual row identifier, thus forming aligned micro-batch pairs.

[0056] For each source micro-batch, the computer device concatenates the event summaries of multiple first-standard data change events in the micro-batch and then performs a hash operation to obtain the first batch summary. Similarly, for the target micro-batch corresponding to the source micro-batch, the computer device concatenates the event summaries of multiple second-standard data change events in the target micro-batch and then performs the same hash operation to obtain the second batch summary. If the first batch summary and the second batch summary are consistent, it indicates that the data records in the source database and the target database corresponding to the micro-batch are generally consistent; if they are inconsistent, it indicates that there are differences within the micro-batch, and it is necessary to further locate the specific difference event. At this time, the computer device can use a binary search method or an event-by-event backtracking method to narrow down the difference range layer by layer within the micro-batch, and finally locate the specific data record with the difference.

[0057] Optionally, in one embodiment, if the first event sequence of the first standard data change event from the source database is consistent with the second event sequence of the second standard data change event from the target database, then an event-by-event comparison is performed on the first standard data change event and the second standard data change event based on the virtual row identifier to obtain the comparison result, including: If the first event sequence of the first standard data change event from the source database is consistent with the second event sequence of the second standard data change event from the target database, then obtain the event delay between the first standard data change event and the second standard data change event corresponding to the same virtual row identifier; If the event delay is less than or equal to the preset delay threshold, the first standard data change event and the second standard data change event are compared event by event based on the virtual row identifier to obtain the comparison result.

[0058] Based on the above embodiments, in addition to considering the consistency of event order, a mechanism for judging event delay is further introduced.

[0059] In this process, after the computer device recognizes that the first event sequence of the first standard data change event from the source database is consistent with the second event sequence of the second standard data change event from the target database, it does not directly perform an event-by-event comparison. Instead, it first obtains the event delay between the first standard data change event and the second standard data change event corresponding to the same virtual row identifier. This event delay represents the difference between the time point when the computer device obtains the first standard data change event and the time point when it obtains the corresponding second standard data change event.

[0060] If the event delay is less than or equal to a preset delay threshold, it indicates that the current network transmission and processing delay is within an acceptable range. In this case, the computer device can perform an event-by-event comparison of the first standard data change event and the second standard data change event based on the virtual row identifier. If the event delay is greater than the preset delay threshold, it indicates that there may be abnormal delay in the current network or processing stage. In this case, the computer device can use a micro-batch comparison method for comparison. Please refer to the relevant descriptions in the above embodiments for details, which will not be repeated here.

[0061] Optionally, in one embodiment, the first data change event and the second data change event contain the primary key values ​​of the data records that have been changed. Following the same allocation strategy, virtual row identifiers are assigned to the first standard data change event and the second standard data change event, respectively, including: Perform a hash operation on the primary key value in the first standard data change event to generate a virtual row identifier for the first standard data change event; Perform the same hash operation on the primary key value in the second standard data change event to generate a virtual row identifier for the second standard data change event.

[0062] In this embodiment, when the logical row identifiers of the first and second data change events are the primary key values ​​of the data records that have been changed, the computer device can, on the one hand, perform a hash operation on the primary key value in the first standard data change event and use the calculated hash value as the virtual row identifier of the first standard data change event; on the other hand, the computer device can perform the same hash operation on the primary key value in the second standard data change event and use the calculated hash value as the virtual row identifier of the second standard data change event. Because hash operations have deterministic and uniformly distributed characteristics, for the same primary key value of a data record, regardless of whether it appears in events in the source database or the target database, a consistent virtual row identifier will be obtained after the same hash operation, thus providing a stable and efficient identifier basis for subsequent comparison operations.

[0063] Optionally, in one embodiment, after comparing the first standard data change event and the second standard data change event based on the virtual row identifier and obtaining the comparison result, the heterogeneous database consistency verification method provided in this application further includes: If the comparison results indicate that the target database and the source database are inconsistent, then based on the data differences between the target database and the source database, an executable database operation statement for eliminating the data differences is generated and output.

[0064] In this embodiment of the application, after the computer device completes the comparison of the first standard data change event and the second standard data change event and obtains the comparison result, if the comparison result indicates that there is a data inconsistency between the target database and the source database, the computer device can further analyze the specific data differences and automatically generate corresponding executable database operation statements to eliminate the data differences between the target database and the source database, thereby ensuring the eventual consistency of the data in the two databases.

[0065] In practice, computer equipment can generate a difference report based on the comparison results. This difference report can list in detail the differences in specific field values ​​of each inconsistent data record in the source database and the target database, as well as the corresponding executable database operation statements.

[0066] Optionally, in one embodiment, based on the data differences between the target database and the source database, executable database operation statements for eliminating data differences are generated and output, including: If the data difference is that the data value corresponding to the same data record in the target database and the source database is different, then an update operation statement is generated to update the data value of the data record corresponding to the same primary key value in the target database to the data value of the data record corresponding to the same primary key value in the source database. If the data difference is a missing data record in the target database, then an insert operation statement is generated to insert the corresponding data record from the source database into the target database. If the data difference is that there are redundant data records in the target database, a delete operation statement is generated to delete the redundant data records from the target database.

[0067] In this embodiment, the computer device can accurately generate corresponding executable database operation statements based on the specific type of data differences, thereby achieving automated repair of the target database.

[0068] Specifically, when the data difference manifests as a discrepancy between the target database and the source database for the same data record, the computer generates an update operation statement to update the data value in the target database to the correct value in the source database. When the data difference manifests as a missing data record in the target database, the computer generates an insert operation statement to insert the corresponding complete data record from the source database into the target database. When the data difference manifests as a redundant data record in the target database that does not exist in the source database, the computer generates a delete operation statement to delete the redundant data record in the target database.

[0069] By employing the differentiated processing strategies described above, precise data repair can be achieved for different scenarios, thereby minimizing human intervention and improving the eventual consistency of data between the two databases.

[0070] Furthermore, another exemplary embodiment describes consistency verification in a database synchronization scenario.

[0071] Scenario: During the replacement of its core transaction system database, a financial institution built a real-time one-way synchronization link from the source database (Oracle 19c) to the target database (YashanDB v23) to ensure business continuity and data consistency. 24 / 7 uninterrupted consistency checks are required on the critical ACCOUNT_TRANS transaction log table (approximately 80 million rows). During peak business periods, the table's update TPS can reach 30,000.

[0072] Deployment and Initialization Deploy Change Data Capture Agent Component A in the Oracle database and Change Data Capture Agent Component B in the YashanDB database.

[0073] Real-time incremental verification process Change occurred: A transfer transaction updated the row with id=500001 in the ACCOUNT_TRANS table of the Oracle database, changing the status from 'PROCESSING' to 'SUCCESS' and the amt field value from 100.00 to 100.50.

[0074] Log capture: Change data capture agent component A captured this update operation from the Oracle database's REDO log, obtaining the complete before image {id:500001,status:'PROCESSING',amt:100.00} and after image {status:'SUCCESS',amt:100.50}.

[0075] Almost simultaneously, the data synchronization tool applies this change to the YashanDB database. Change data capture agent component B also captures the corresponding update operation and its data image from the YashanDB database's REDO logs.

[0076] Semantic normalization: The VARCHAR2 type status and NUMBER type amt from standard data change events in the Oracle database are mapped to the standard utility types CANONICAL_STRING and CANONICAL_DECIMAL, respectively. The virtual row number VRID_Ora_500001 is calculated, and digests Hash_old_O and Hash_new_O are generated from the normalized values ​​of the before and after mappings.

[0077] The VARCHAR and DECIMAL types from standard data change events in the YashanDB database are mapped to the same CANONICAL_STRING and CANONICAL_DECIMAL. Since the primary key values ​​are the same, the same virtual row number VRID_Yas_500001 is calculated (equal to VRID_Ora_500001). Digests Hash_old_Y and Hash_new_Y are generated for the captured values.

[0078] Streaming comparison: Events from both ends were matched, and a comparison was made: Hash_old_O == Hash_old_Y (indicating the state was consistent before the update), and Hash_new_O == Hash_new_Y (indicating the state was consistent after the update). The data in this row was determined to be consistent, and the process ended.

[0079] Furthermore, suppose that due to a temporary malfunction of the synchronization tool, the above update to id=500001 failed to be successfully synchronized to the YashanDB database.

[0080] Comparison process: The computer device obtains data change events from the Oracle database and performs semantic normalization to obtain a standard data change event A (VRID, Hash_old_O, Hash_new_O).

[0081] Within the set time window, no database data change events were received from the YashanDB database, and correspondingly, there was no standard data change event with the same virtual row number as standard data change event A.

[0082] The computer device then triggered a difference alert for "target update missing".

[0083] Repair generation: Based on its configuration, the computer device retrieves the complete post-image data of the event (status='SUCCESS', amt=100.50) from the cache of the change data capture agent component A, and generates a semantically correct database operation statement for the YashanDB database: UPDATEACCOUNT_TRANSSETstatus='SUCCESS',amt=100.50WHEREid=500001; The database operation statements and difference details are recorded together in the difference report, which can be used for manual review or to drive automated repair processes.

[0084] To facilitate better implementation of the heterogeneous database consistency verification method described above, this application also provides a corresponding heterogeneous database consistency verification device. The meanings of the terms used are the same as in the heterogeneous database consistency verification method described above; for specific implementation details, please refer to the descriptions in the above method embodiments.

[0085] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a heterogeneous database consistency verification device provided in an embodiment of this application. The heterogeneous database consistency verification device may include an event acquisition module 210, a specification processing module 220, an identifier allocation module 230, and a data verification module 240, wherein... The event acquisition module 210 is used to parse the transaction logs of the source database and the target database in read-only mode to obtain the first data change event of the source database and the second data change event of the target database. The standardization processing module 220 is used to perform semantic standardization processing on the data in the first data change event and the second data change event according to the same standardization strategy, so as to obtain the first standard data change event and the second standard data change event. The identifier allocation module 230 is used to assign virtual row identifiers to the change description data in the first standard data change event and the second standard data change event according to the same allocation strategy. The data verification module 240 is used to compare the first standard data change event and the second standard data change event based on the virtual row identifier to obtain the comparison result.

[0086] Optionally, in one embodiment, the data verification module 240 is used to perform an event-by-event comparison of the first standard data change event and the second standard data change event based on the virtual row identifier when the first event order of the first standard data change event from the source database and the second event order of the second standard data change event from the target database are consistent, and obtain a comparison result; or, when the first event order and the second event order are inconsistent, the first standard data change event and the second standard data change event are organized into micro-batches based on the virtual row identifier to obtain a source micro-batch and a target micro-batch; calculate the first batch summary of the source micro-batch and the second batch summary of the target micro-batch; compare the first batch summary and the second batch summary to obtain a comparison result.

[0087] Optionally, in one embodiment, the data verification module 240 is used to obtain the event delay between the first standard data change event and the second standard data change event corresponding to the same virtual row identifier when the first event sequence of the first standard data change event from the source database and the second event sequence of the second standard data change event from the target database are consistent; when the event delay is less than or equal to a preset delay threshold, the first standard data change event and the second standard data change event are compared event by event based on the virtual row identifier to obtain the comparison result.

[0088] Optionally, in one embodiment, the standardization processing module 220 is used to map the data types in the first data change event and the second data change event to a unified standard data type; and to convert the data values ​​in the first data change event and the second data change event into unified standard data values.

[0089] Optionally, in one embodiment, the change description data includes the primary key value of the data record that has been changed. According to the same allocation strategy, the identifier allocation module 230 is used to perform a hash operation on the primary key value contained in the change description data in the first standard data change event to generate a virtual row identifier for the change description data in the first standard data change event; and to perform the same hash operation on the primary key value contained in the change description data in the second standard data change event to generate a virtual row identifier for the change description data in the second standard data change event.

[0090] Optionally, in one embodiment, the heterogeneous database consistency verification device provided in this application further includes a difference elimination module, which is used to generate and output executable database operation statements for eliminating data differences based on the data differences between the target database and the source database when the comparison result indicates that the target database and the source database are inconsistent.

[0091] Optionally, in one embodiment, the difference exclusion module is used to generate an update operation statement to update the data value of the data record corresponding to the same primary key value in the target database to the data value of the data record corresponding to the same primary key value in the source database when the data difference is that the data value corresponding to the same primary key value in the target database is different; when the data difference is that the data record is missing in the target database, it generates an insert operation statement to insert the corresponding data record in the source database into the target database; if the data difference is that there is redundant data record in the target database, it generates a delete operation statement to delete the redundant data record from the target database.

[0092] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0093] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 4 As shown, the computer device includes: at least one processor ( Figure 4 The diagram shows only one instance of a memory and a computer program stored in the memory that can run on at least one processor. When the processor executes the computer program, it implements the steps in the above-described heterogeneous database consistency verification method embodiment.

[0094] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0095] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0096] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0098] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a computer device, it enables the computer device to execute the steps in the above method embodiments.

[0099] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0100] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A consistency verification method for heterogeneous databases, characterized in that, include: The transaction logs of the source database and the target database are parsed in read-only mode to obtain the first data change event of the source database and the second data change event of the target database. Following the same standard strategy, semantic normalization processing is performed on the data in the first data change event and the second data change event respectively to obtain the first standard data change event and the second standard data change event; Virtual row identifiers are assigned to the first standard data change event and the second standard data change event respectively, using the same allocation strategy; Based on the virtual row identifier, the first standard data change event and the second standard data change event are compared to obtain the comparison result.

2. The consistency verification method for heterogeneous databases according to claim 1, characterized in that, The comparison of the first standard data change event and the second standard data change event based on the virtual row identifier to obtain the comparison result includes: If the first event sequence of the first standard data change event from the source database is consistent with the second event sequence of the second standard data change event from the target database, then the first standard data change event and the second standard data change event are compared event by event based on the virtual row identifier to obtain the comparison result; If the order of the first event and the order of the second event are inconsistent, the first standard data change event and the second standard data change event are organized into micro-batches based on the virtual row identifier to obtain source micro-batches and target micro-batches; the first batch summary of the source micro-batches and the second batch summary of the target micro-batches are calculated; the first batch summary and the second batch summary are compared to obtain the comparison result.

3. The consistency verification method for heterogeneous databases according to claim 2, characterized in that, If the first event sequence of the first standard data change event from the source database is consistent with the second event sequence of the second standard data change event from the target database, then an event-by-event comparison is performed on the first standard data change event and the second standard data change event based on the virtual row identifier to obtain the comparison result, including: If the first event sequence of the first standard data change event from the source database is consistent with the second event sequence of the second standard data change event from the target database, then the event delay between the first standard data change event and the second standard data change event corresponding to the same virtual row identifier is obtained. If the event delay is less than or equal to a preset delay threshold, then the first standard data change event and the second standard data change event are compared event by event based on the virtual row identifier to obtain the comparison result.

4. The consistency verification method for heterogeneous databases according to claim 1, characterized in that, The step of performing semantic normalization processing on the data in the first data change event and the second data change event respectively to obtain the first standard data change event and the second standard data change event includes: Map the data types in the first data change event and the second data change event to a unified standard data type; The data values ​​in the first data change event and the second data change event are converted into unified standard data values ​​to obtain the first standard data change event and the second standard data change event.

5. The consistency verification method for heterogeneous databases according to claim 1, characterized in that, The first data change event and the second data change event contain the primary key values ​​of the data records that have been changed. The step of assigning virtual row identifiers to the first standard data change event and the second standard data change event according to the same allocation strategy includes: Perform a hash operation on the primary key value in the first standard data change event to generate a virtual row identifier for the first standard data change event; Perform the same hash operation on the primary key value in the second standard data change event to generate a virtual row identifier for the second standard data change event.

6. The consistency verification method for heterogeneous databases according to claim 1, characterized in that, After comparing the first standard data change event and the second standard data change event based on the virtual row identifier to obtain the comparison result, the method further includes: If the comparison result indicates that the target database and the source database are inconsistent, then an executable database operation statement for eliminating the data difference is generated and output based on the data difference between the target database and the source database.

7. The consistency verification method for heterogeneous databases according to claim 1, characterized in that, Based on the data differences between the target database and the source database, generate and output executable database operation statements for eliminating the data differences, including: If the data difference is that the data value corresponding to the same data record in the target database and the source database is different, then an update operation statement is generated to update the data value of the data record corresponding to the same primary key value in the target database to the data value of the data record corresponding to the same primary key value in the source database; If the data difference is a missing data record in the target database, then an insert operation statement is generated to insert the corresponding data record in the source database into the target database; If the data difference indicates that there are redundant data records in the target database, a deletion operation statement is generated to delete the redundant data records from the target database.

8. A consistency verification device for heterogeneous databases, characterized in that, include: The event acquisition module is used to parse the transaction logs of the source database and the target database in read-only mode to obtain the first data change event of the source database and the second data change event of the target database. The standardization processing module is used to perform semantic standardization processing on the data in the first data change event and the second data change event respectively according to the same standardization strategy to obtain the first standard data change event and the second standard data change event. The identifier allocation module is used to allocate virtual row identifiers to the first standard data change event and the second standard data change event according to the same allocation strategy; The data verification module is used to compare the first standard data change event and the second standard data change event based on the virtual row identifier to obtain the comparison result.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that can run on the processor, and the processor, when running the computer program, implements the heterogeneous database consistency verification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the heterogeneous database consistency verification method as described in any one of claims 1 to 7.

Images