A database synchronization verification method and device based on data storage and a medium

Abstract

The application discloses a kind of database synchronization verification methods, equipment and medium based on data storage, it is related to synchronous verification technical field, including, obtain database synchronization task, read the storage information of source database and target database, obtain source storage information and target storage information, respectively extract generation element from source storage information and target storage information, obtain source end generation information and target end generation information, and carry out corresponding verification, obtain anchoring result, according to anchoring result in source storage information and target storage information Determine the verification range under the same storage generation, split data content element and storage attribution element from the source storage information in verification range;The application can avoid directly judging content when the verification object has not been aligned by limiting the subsequent verification range in the data range that has completed synchronization and can be read, thereby reducing the misjudgment caused by synchronization not being completed, transaction invisible, storage state change and the like.

Description

Technical Field

[0001] This invention relates to the field of synchronization verification technology, and in particular to a database synchronization verification method, device and medium based on data storage. Background Technology

[0002] With the development of big data services, distributed databases, cloud storage architectures, and multi-replica disaster recovery technologies, database synchronization has evolved from single master-slave replication to a cross-database, cross-region, and cross-storage tier data collaboration process. The source and target databases typically need to maintain consistency in synchronization batches, transaction commit order, table structure versions, and data page changes to ensure business read / write continuity and disaster recovery reliability. Existing database synchronization verification technologies often employ methods such as record count comparison, field value comparison, log replay verification, checksum comparison, or digest value comparison to determine whether synchronization discrepancies exist between the source and target databases.

[0003] However, existing database synchronization verification technology still has room for improvement. Firstly, it easily overlooks factors such as inconsistent synchronization batches, inconsistent table structure versions, inconsistent data page versions, changes in partition ownership, storage layer migration, and transactions that are not yet visible. Secondly, data that has not yet been synchronized or has not yet entered a readable state is easily misjudged as having content differences. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a database synchronization verification method based on data storage to solve the problems of easily overlooked factors such as inconsistent synchronization batches, inconsistent table structure versions, inconsistent data page versions, changes in partition ownership, storage layer migration, and transactions that are not yet visible.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a database synchronization verification method based on data storage, comprising, Obtain the database synchronization task, read the storage information of the source database and the target database, and obtain the source storage information and the target storage information; Extract generational elements from the source storage information and the target storage information respectively to obtain source generational information and target generational information, and perform corresponding verification to obtain anchoring results; Based on the anchoring results, determine the verification range under the same storage generation in the source storage information and the target storage information. From the source storage information within the verification range, split the data content elements and storage ownership elements, save the data content elements as the source logical root, and save the storage ownership elements as the source physical root. Using the source logical root and source physical root as verification benchmarks, locate the data content elements and storage ownership elements in the target stored information, and organize them into the target logical root and target physical root. The correspondence between the data content of the source logical root and the target logical root is verified layer by layer to form a logical result. The storage location relationship of the target physical root is checked along the storage ownership path of the source physical root to form a physical result. Read the logical and physical results, classify them in parallel to form a dual-root result, determine the content difference, storage difference, and mixed difference based on the dual-root result, and generate the verification result.

[0007] As a preferred embodiment of the database synchronization verification method based on data storage described in this invention, the database synchronization task includes a source database identifier, a target database identifier, a data table identifier, a synchronization batch identifier, a synchronization direction identifier, and a verification trigger identifier. The source storage information includes source data content elements, source storage ownership elements, and source generation elements; The target-end storage information includes target-end data content elements, target-end storage ownership elements, and target-end generation elements.

[0008] As a preferred embodiment of the database synchronization verification method based on data storage described in this invention, the step of obtaining the source-end generation information and the target-end generation information specifically involves: Read the source-end synchronization batch identifier, source-end table structure version, source-end transaction commit boundary, source-end transaction visibility boundary, source-end data page version, source-end partition version, and source-end storage layer identifier from the source-end storage information, arrange and save them according to the generation record format, and obtain the source-end generation information. Read the target-side synchronization batch identifier, target-side table structure version, target-side transaction commit boundary, target-side transaction visibility boundary, target-side data page version, target-side partition version, and target-side storage layer identifier from the target-side storage information, arrange them according to the generation record format, and save them to obtain the target-side generation information.

[0009] As a preferred embodiment of the database synchronization verification method based on data storage described in this invention, the step of performing corresponding verification to obtain the anchoring result specifically includes: Read the source generation information and target generation information. Using the synchronization batch identifier in the database synchronization task as the verification benchmark, match the source synchronization batch identifier with the target synchronization batch identifier, match the source table structure version with the target table structure version, match the source data page version with the target data page version, match the source partition version with the target partition version, and match the source storage layer identifier with the target storage layer identifier to obtain the generation correspondence record. Read the source transaction commit boundary and the source transaction visibility boundary to determine the source verifiable boundary. Read the target transaction commit boundary and the target transaction visibility boundary to determine the target verifiable boundary. Mark the overlapping part of the source verifiable boundary and the target verifiable boundary as the generation verifiable range. When the generation correspondence record represents the batch correspondence, structure correspondence, data page correspondence, partition correspondence and storage layer correspondence, and the generation verifiable range exists, the source generation information and the target generation information are marked as the same storage generation, and the anchoring result is obtained. When there are uncorresponding entries in the generation-corresponding records and the generation verifiable range does not exist, the source generation information and the target generation information are marked as different storage generations, resulting in an anchoring failure. The anchoring result is either the anchoring success result or the anchoring failure result.

[0010] As a preferred embodiment of the database synchronization verification method based on data storage described in this invention, wherein: saving data content elements as source-end logical roots and saving storage attribution elements as source-end physical roots specifically refers to: Read the anchoring results. When the anchoring results include anchoring success results, mark the data range in the verifiable range corresponding to the data table identifier as the verification range. Within the verification range, read the source storage information and classify the source primary key identifier, source field content, source field sequence, source row version identifier, and source deletion status identifier into the data content elements. Classify the source data page identifier, source partition identifier, source file fragment identifier, source compressed block identifier, and source storage layer identifier into the storage ownership elements. The data content elements are arranged according to the primary key identifier and the order of the source field, and the arranged data content elements are saved as the source logical root. The storage ownership elements are arranged according to the hierarchical relationship of source data page identifier, source partition identifier, source file fragment identifier, source compressed block identifier, and source storage layer identifier, and the arranged storage ownership elements are saved as the source physical root.

[0011] As a preferred embodiment of the database synchronization verification method based on data storage described in this invention, the step of organizing the data into a target logical root and a target physical root specifically includes: Read the source-end primary key identifier, source-end field sequence, source-end row version identifier, and source-end deletion status identifier from the source-end logical root to form a data content index; Read the source data page identifier, source partition identifier, source file fragment identifier, source compressed block identifier, and source storage layer identifier from the source physical root to form a storage ownership index; By using data content indexing, the target primary key identifier, target field content, target field ordinal, target row version identifier, and target deletion status identifier are located in the target storage information within the verification range to obtain the target data content elements. Based on the storage ownership index, locate the target data page identifier, target partition identifier, target file fragment identifier, target compressed block identifier, and target storage layer identifier in the target storage information within the verification range to obtain the target storage ownership elements; Organize the target data content elements according to the order in the data content index and save them as the target logical root; Organize the target storage ownership elements according to the hierarchical relationship in the storage ownership index and save them as the target physical root.

[0012] As a preferred embodiment of the database synchronization verification method based on data storage described in this invention, the formation of the physical result specifically refers to: Read the source logical root and the target logical root, match the data table identifier in the source logical root with the data table identifier in the target logical root, and obtain the corresponding status of the data table; When the corresponding status of the data table indicates that the data table correspondence is true, the primary key identifier in the source logical root is used as the retrieval basis to find the corresponding primary key identifier in the target logical root to obtain the primary key corresponding status. When the primary key correspondence status indicates that the primary key correspondence is true, the field content in the source logical root and the field content in the target logical root are read sequentially according to the field order, and the field order and field content are checked item by item to obtain the field correspondence status. When the corresponding status of the fields is true, the row version identifier in the source logical root and the row version identifier in the target logical root are reviewed, and the deletion status identifier in the source logical root and the deletion status identifier in the target logical root are also reviewed to obtain the corresponding status of the version deletion. Write the corresponding status of the data table, the corresponding status of the primary key, the corresponding status of the field, and the corresponding status of the version deletion into the data content review record. Mark the non-corresponding items in the data content review record as differences, mark the corresponding items in the data content review record as consistency, and save the difference marks and consistency marks as the data content consistency status. Save the data content consistency status as a logical result. Read the source physical root and the target physical root. Read the storage layer identifier, file fragment identifier, compressed block identifier, partition identifier and data page identifier in the source physical root. Sort them according to the subordinate order of storage layer identifier, file fragment identifier, compressed block identifier, partition identifier and data page identifier, and concatenate the sorted identifiers to save as the storage home path. The storage location relationship is obtained by sequentially checking the storage layer identifier, file fragment identifier, compressed block identifier, partition identifier, and data page identifier in the physical root of the target end along the storage ownership path. Mark the corresponding and non-corresponding items in the storage location relationship as consistent items and misaligned items respectively, and save them as storage ownership consistency status. Save the storage ownership consistency status as physical result.

[0013] As a preferred embodiment of the database synchronization verification method based on data storage described in this invention, the generation of verification results specifically includes: Read the logical and physical results, identify consistent and inconsistent items in the logical results, and identify consistent and misaligned items in the physical results; The consistent and differing items in the logical results are arranged side by side with the consistent and misaligned items in the physical results, and the parallel arrangement results are categorized and marked to form a double root result; When the double root result contains only consistent items, it is determined to be synchronously consistent; when the double root result contains differing items but not misaligned items, it is determined to be content differing; when the double root result does not contain differing items but contains misaligned items, it is determined to be storage differing; when the double root result contains both differing items and misaligned items, it is determined to be mixed differing. Read the data table identifier, primary key identifier, field ordinal number, row version identifier, deletion status identifier, data page identifier, partition identifier, file fragment identifier, compressed block identifier, and storage layer identifier corresponding to the double root result, and organize them into a verification detail; Save the double root result and verification details as the verification result.

[0014] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the database synchronization verification method based on data storage as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the database synchronization verification method based on data storage as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By confirming that the source and target databases are comparable before performing data content comparison, and limiting the subsequent verification scope to the data that has been synchronized and can be read, it avoids directly judging content when the verification objects are not yet aligned, thereby reducing misjudgments caused by incomplete synchronization, transaction invisibility, or changes in storage status. The verification results no longer simply reflect whether field values ​​are consistent, but rather more realistically reflect the synchronization status between the source and target databases, making anomaly detection more accurate, difference investigation more focused, and subsequent synchronization compensation and maintenance more targeted, ultimately improving the accuracy, stability, and reliability of database synchronization verification results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a database synchronization verification method based on data storage.

[0019] Figure 2 A flowchart is created to anchor the results.

[0020] Figure 3 Organize the process for logical roots and physical roots.

[0021] Figure 4 This is the process for verifying the results of double root determination. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a database synchronization verification method based on data storage, including the following steps: S1: Obtain the database synchronization task, read the storage information of the source database and the target database, and obtain the source storage information and the target storage information.

[0026] The source database is determined according to the source database identifier in the database synchronization task; the target database is determined according to the target database identifier; the range of data tables to be verified is determined according to the data table identifier; the synchronization batch to be verified is determined according to the synchronization batch identifier; and the verification direction from the source database to the target database is determined according to the synchronization direction identifier. When the verification trigger identifier indicates that verification is required, the source data content elements, source storage ownership elements, and source generation elements corresponding to the data table identifier and synchronization batch identifier are read from the source database's data dictionary table, transaction log, page directory, storage engine metadata, and synchronization log. When the source data page identifier, source compressed block identifier, or source file fragment identifier cannot be directly... When reading from the source database, corresponding identifiers are generated from the page directory, compressed block directory, and file fragment mapping records of the source database according to preset acquisition rules to obtain source storage information. From the data dictionary table, transaction log, page directory, storage engine metadata, and synchronization log of the target database, the target data content elements, target storage ownership elements, and target generation elements corresponding to the data table identifier and synchronization batch identifier are read. When the target data page identifier, target compressed block identifier, or target file fragment identifier cannot be directly read from the target database, corresponding identifiers are generated from the page directory, compressed block directory, and file fragment mapping records of the target database according to preset acquisition rules to obtain target storage information.

[0027] To further explain, the data acquisition rules are based on the pre-configured storage engine type, data table storage structure, page directory organization method, compressed block organization method, file fragment mapping method, and synchronization batch identifier of the source and target databases. The acquisition rules include: when the data page identifier cannot be directly read, reading the data table identifier, page start address, page end address, and page sequence number from the page directory, and generating a data page identifier in the order of data table identifier, page start address, page end address, and page sequence number; when the compressed block identifier cannot be directly read, reading the compressed block start address, compressed block end address, compression algorithm identifier, and compression sequence number from the compressed block directory. The compressed block sequence number is generated, and a compressed block identifier is generated in the order of compressed block start address, compressed block end address, compression algorithm identifier, and compressed block sequence number. When the file fragment identifier cannot be read directly, the storage file identifier, fragment start address, fragment length, and fragment partition identifier in the file fragment mapping record are read, and a file fragment identifier is generated in the order of storage file identifier, fragment start address, fragment length, and fragment partition identifier. The generated data page identifier, compressed block identifier, and file fragment identifier are associated and saved with the data table identifier and synchronization batch identifier as the corresponding identifiers in the source acquisition table or the target acquisition table.

[0028] S2: Extract generational elements from the source storage information and the target storage information respectively to obtain the source generational information and the target generational information, and perform corresponding verification to obtain the anchoring result.

[0029] S2.1: After obtaining the source and target storage information, according to the data table identifier and synchronization batch identifier in the database synchronization task, read the source synchronization batch identifier, source table structure version, source transaction commit boundary, source transaction visibility boundary, source data page version, source partition version, and source storage layer identifier from the source generation elements in the source storage information. Specifically, the source data page version is read from the page directory, page header version record, or page change log of the source database; the source partition version is read from the partition metadata or partition change log of the source database; and the source storage layer identifier is read from the storage index of the source database. Read from metadata, tablespace records, or storage layer mapping records; when the source data page version, source partition version, or source storage layer identifier cannot be directly read, generate the page directory change record, partition change record, and storage layer mapping record by associating them according to the synchronization batch identifier and data table identifier. Then, save the generational record format according to the generational record format of synchronization batch identifier, table structure version, transaction commit boundary, transaction visibility boundary, data page version, partition version, and storage layer identifier. The generational record format is a metadata record saved according to the field name and value correspondence. The field names include synchronization batch identifier, table structure version, transaction commit boundary, and transaction visibility boundary. The target generation information is obtained by retrieving the boundary, transaction visibility boundary, data page version, partition version, and storage layer identifier, with the field values ​​being the corresponding read or generated identifier values. Based on the same data table identifier and the same synchronization batch identifier, the target generation elements in the target storage information are retrieved: target synchronization batch identifier, target table structure version, target transaction commit boundary, target transaction visibility boundary, target data page version, target partition version, and target storage layer identifier. Specifically, the target data page version is retrieved from the target database's page directory, page header version record, or page change log, and the target partition version is retrieved from the target database's page directory, page header version record, or page change log. The target storage layer identifier is read from the partition metadata or partition change log of the target database, and from the storage engine metadata, tablespace records, or storage layer mapping records of the target database. When the target data page version, target partition version, or target storage layer identifier cannot be read directly, the page directory change record, partition change record, and storage layer mapping record are associated and generated according to the synchronization batch identifier and data table identifier, and arranged and saved in the same generation record format to obtain the target generation information. This ensures that the source generation information and the target generation information have the same field order and the same verification dimension for subsequent corresponding verification.

[0030] S2.2: Read the source and target generation information. Using the synchronization batch identifier in the database synchronization task as the verification benchmark, locate the source synchronization batch identifier, source table structure version, source data page version, source partition version, and source storage layer identifier in the source generation information in sequence. Similarly, locate the target synchronization batch identifier, target table structure version, target data page version, target partition version, and target storage layer identifier in the target generation information in sequence. Perform a field value consistency check between the source and target synchronization batch identifiers. If the field values ​​are the same, write "batch correspondence is established" to the corresponding batch bit. When field values ​​differ, record the batch mismatch in the corresponding batch location; perform field value consistency and version compatibility checks between the source table structure version and the target table structure version; if field values ​​are the same or the preset version mapping indicates version compatibility, record the structure mismatch in the corresponding structure location; if field values ​​differ and the preset version mapping does not indicate version compatibility, record the structure mismatch in the corresponding structure location; perform field value consistency and version compatibility checks between the source data page version and the target data page version, and write the results to the corresponding data page location; and match the source partition version with the target partition version. The system performs field value consistency and version compatibility checks on the version, writing the results to the corresponding location in the partition. It then performs field value consistency and storage layer mapping consistency checks on the source and target storage layer identifiers, writing the results to the corresponding storage layer location to obtain the generation-corresponding record. Next, it reads the source transaction commit boundary and source transaction visibility boundary. Transaction segments are represented by segment start and end boundary values, which are the transaction commit sequence number, log sequence number, and commit timestamp—each of these values ​​must be monotonically increasing in both the source and target databases. Finally, it commits the source transaction. Transaction segments that have been committed and fall within the visible boundary of the source transaction are saved as verifiable boundaries of the source. The transaction commit boundary and the visible boundary of the target transaction are read. Transaction segments that have been committed and fall within the visible boundary of the target transaction are saved as verifiable boundaries of the target. Transaction segments whose start and end boundary values ​​overlap in the verifiable boundaries of the source and target are marked as verifiable ranges. This ensures that subsequent verification is performed only within the range where both the source and target databases have committed and can be read, preventing unseen synchronized content from being misjudged as content differences.

[0031] To further explain, the version mapping relationship is pre-established based on table structure change records, page structure change records, partition change records, and storage layer change records already recorded before the database synchronization task is executed. The version mapping relationship includes equi-mapping and compatibility mapping. Equi-mapping indicates that the source version identifier and the target version identifier are completely identical. Compatibility mapping indicates that although the source and target version identifiers are different, the field arrangement, field type, page layout, partition boundaries, and storage level corresponding to the target version identifier can accommodate the storage content corresponding to the source version identifier. When the source table... If the source and target table structure versions correspond to the same set of fields and field order in the table structure change record, they are considered version compatible. If the source and target data page versions correspond to the same page capacity, page header format, and row positioning method in the page structure change record, they are considered version compatible. If the source and target partition versions correspond to the same partition boundary and partition affiliation rule in the partition change record, they are considered version compatible. If the source and target storage layer identifiers correspond to the same storage layer and the same placement rule in the storage layer change record, they are considered to have consistent mapping.

[0032] S2.3: Read the batch correspondence, structure correspondence, data page correspondence, partition correspondence, and storage layer correspondence from the generation correspondence record, and read the generation verifiable range; when the batch correspondence, structure correspondence, data page correspondence, partition correspondence, and storage layer correspondence are all in a valid state, and the generation verifiable range record contains a transaction segment that is readable by both the source database and the target database, mark the source generation information and the target generation information as the same storage generation, and obtain the anchoring valid result. The anchoring valid result indicates that the source database and the target database are not only in the same synchronization batch, the same table structure version, the same data page version, the same partition version, and the same storage layer identifier, but also have common readable characteristics. The transaction basis enables subsequent verification scopes to exclude synchronized content that has not yet been committed or entered a visible state, thereby reducing false judgments caused by synchronization delays. When there are uncorresponding entries in the generation-corresponding records, and the generation-verifiable scope does not record a transaction segment that can be read by both the source and target databases, the source and target generation information are marked as different storage generations, resulting in an anchoring failure result. The anchoring failure result indicates that the source and target databases do not have a verification basis under the same storage generation. One of the anchoring success result and the anchoring failure result is saved as the anchoring result, so that the anchoring result can directly limit whether the subsequent verification scope continues to enter the double root verification.

[0033] Preferably, compared to existing methods that directly compare field content, record quantity, or summary value, this invention first performs generational mapping on synchronization batch, table structure version, data page version, partition version, and storage layer identifier. Then, it determines the verifiable range of generations through transaction commit boundaries and transaction visibility boundaries. This ensures that verification is only performed within transaction segments where both the source and target databases have committed and can be read, avoiding misjudgments caused by synchronization delays, uncommitted transactions, or unvisible transactions, thereby improving the accuracy and stability of database synchronization verification.

[0034] S3: Based on the anchoring results, determine the verification range under the same storage generation in the source storage information and the target storage information. From the source storage information within the verification range, split the data content elements and storage ownership elements, save the data content elements as the source logical root, and save the storage ownership elements as the source physical root.

[0035] S3.1: Read the anchoring result. When the anchoring result contains an anchoring success result, read the generationally verifiable range from the anchoring result. Based on the data table identifier in the database synchronization task, search for the source and target data ranges belonging to the same data table identifier within the generationally verifiable range. Using the source data range as a reference, map the source data range to the target data range according to the data table identifier, synchronization batch identifier, and transaction segment boundary. Mark the intersection range where both the source and target data ranges exist and fall within the generationally verifiable range. Verification Scope: Within the verification scope, continue reading source storage information. Include the source primary key identifier, source field content, source field sequence, source row version identifier, and source deletion status identifier, which represent record content and record status, into the data content elements. Include the source data page identifier, source partition identifier, source file fragment identifier, source compressed block identifier, and source storage layer identifier, which represent record storage location and storage level, into the storage ownership elements. Use the data content elements to form the source logical root and the storage ownership elements to form the source physical root.

[0036] S3.2: Locate the source field content, source field sequence, source row version identifier, and source deletion status identifier in each data content element according to the source primary key identifier. Under the same source primary key identifier, arrange the source field content from front to back according to the source field sequence to ensure that the field content in the same record has a fixed order. Then save the arranged source primary key identifier, source field content, source field sequence, source row version identifier, and source deletion status identifier as the source logical root. Arrange the storage belonging elements according to the hierarchical relationship of source storage layer identifier, source file fragment identifier, source compressed block identifier, source partition identifier, and source data page identifier. This ensures that the source storage layer identifier limits the overall storage layer, the source file fragment identifier limits the file fragment position, the source compressed block identifier limits the compressed block position, the source partition identifier limits the partition position, and the source data page identifier limits the data page position. Then save the arranged source data page identifier, source partition identifier, source file fragment identifier, source compressed block identifier, and source storage layer identifier as the source physical root.

[0037] S4: Using the source logical root and source physical root as verification benchmarks, locate the data content elements and storage ownership elements in the target stored information, and organize them into the target logical root and target physical root.

[0038] S4.1: Read the source-end primary key identifier from the source-end logical root, and under the same source-end primary key identifier, continue to read the source-end field sequence number, source-end row version identifier, and source-end deletion status identifier. Use the source-end primary key identifier as the record location basis, the source-end field sequence number as the field location basis, the source-end row version identifier as the record version location basis, and the source-end deletion status identifier as the record status location basis. Organize and save the data in the order of source-end primary key identifier, source-end field sequence number, source-end row version identifier, and source-end deletion status identifier to form a data content index; read the source-end physical root... The source storage layer identifier, source file fragment identifier, source compressed block identifier, source partition identifier, and source data page identifier are used as the basis for storage layer location, file fragment identifier as the basis for file fragment location, compressed block identifier as the basis for compressed block location, partition identifier as the basis for partition location, and data page identifier as the basis for data page location. These identifiers are organized and saved in hierarchical order to form a storage ownership index.

[0039] S4.2: Based on the source primary key identifier in the data content index, search for the target primary key identifier corresponding to the source primary key identifier in the target storage information within the verification range. After finding the target primary key identifier, search for the corresponding target field sequence based on the source field sequence in the data content index, and read the target field content corresponding to the target field sequence. Simultaneously, based on the source row version identifier and source deletion status identifier in the data content index, read the corresponding target row version identifier and target deletion status identifier. Organize the target primary key identifier, target field content, target field sequence, target row version identifier, and target deletion status identifier into target data content elements. Based on the source storage layer identifier, source file fragment identifier, source compressed block identifier, source partition identifier, and source data page identifier in the storage ownership index, read the pre-established storage ownership mapping table. The storage ownership mapping table is based on the same data table identifier, the same synchronization batch identifier, the same transaction segment boundary, and the same... A primary key identifier is used to associate and store the source storage layer identifier with the target storage layer identifier, the source file fragment identifier with the target file fragment identifier, the source compressed block identifier with the target compressed block identifier, the source partition identifier with the target partition identifier, and the source data page identifier with the target data page identifier. When the identifier value in the source storage ownership index matches the identifier value in the target storage information, a step-by-step search is performed according to the equivalent correspondence. When the identifier value in the source storage ownership index does not match the identifier value in the target storage information, a conversion search is performed according to the association storage relationship in the storage ownership mapping table. The corresponding target storage layer identifier, target file fragment identifier, target compressed block identifier, target partition identifier, and target data page identifier are searched step-by-step in the target storage information within the verification range, and the found target data page identifier, target partition identifier, target file fragment identifier, target compressed block identifier, and target storage layer identifier are organized into target storage ownership elements.

[0040] To further explain, the storage ownership mapping table is pre-established based on the database synchronization task, the storage engine metadata of the source database, the storage engine metadata of the target database, the page directory, partition metadata, file fragment mapping records, compressed block directory, and synchronization log of the source and target databases under the same synchronization batch. The storage ownership mapping table includes source database identifier, target database identifier, data table identifier, synchronization batch identifier, transaction segment boundary, primary key identifier, source storage layer identifier, target storage layer identifier, source file fragment identifier, target file fragment identifier, source compressed block identifier, target compressed block identifier, source partition identifier, target partition identifier, source data page identifier, target data page identifier, and mapping type. The mapping type includes equi-mapping and transformation mapping. Equi-mapping indicates that the source identifier and the target identifier have the same identifier value, while transformation mapping indicates that the source identifier and the target identifier have different identifier values, but point to the same storage content under the same data table identifier, the same synchronization batch identifier, the same transaction segment boundary, and the same primary key identifier.

[0041] S4.3: Following the order of the source primary key identifier, source field ordinal, source row version identifier, and source deletion status identifier in the data content index, reorganize the target primary key identifier, target field ordinal, target row version identifier, and target deletion status identifier in the target data content elements. Place the target field content under its corresponding target field ordinal, ensuring the target data content elements maintain the same order as the data content index. Then, reorganize the reorganized target primary key identifier, target field content, target field ordinal, target row version identifier, and target deletion status identifier. The deletion status identifier of the target end is saved as the logical root of the target end; according to the hierarchical relationship of the source end storage layer identifier, source end file fragment identifier, source end compressed block identifier, source end partition identifier, and source end data page identifier in the storage ownership index, the target end storage layer identifier, target end file fragment identifier, target end compressed block identifier, target end partition identifier, and target end data page identifier in the target end storage ownership element are sorted out accordingly, and then the sorted target end data page identifier, target end partition identifier, target end file fragment identifier, target end compressed block identifier, and target end storage layer identifier are saved as the physical root of the target end.

[0042] S5: Verify the correspondence between the data content of the source logical root and the target logical root layer by layer to form a logical result. Check the storage location relationship of the target physical root along the storage ownership path of the source physical root to form a physical result.

[0043] S5.1: Read the table identifiers in the source logical root and the target logical root, match the field values ​​of the table identifiers in the source logical root and save the matching result as the table matching status; when the table matching status indicates that the table identifiers are consistent, continue to read the primary key identifier in the source logical root, and use the primary key identifier in the source logical root as the retrieval basis to search for the primary key identifier in the target logical root that is consistent with the primary key identifier in the source logical root; when there is a primary key identifier in the target logical root that is consistent with the primary key identifier in the source logical root, save the primary key matching status; when there is no primary key identifier in the target logical root that is consistent with the primary key identifier in the source logical root, save the primary key matching status.

[0044] S5.2: When the primary key correspondence status indicates that the primary key correspondence is true, read the field content in the source logical root sequentially according to the field order in the source logical root, and read the field content corresponding to the same field order in the target logical root. Verify the field content in the source logical root and the field content in the target logical root under the same field order item by item. When the field order and field content are consistent, save the field correspondence as true as the field correspondence status. When there is a missing field order, inconsistent field order, or inconsistent field content, save the field correspondence as false as the field correspondence status. When the field correspondence status indicates that the field correspondence is true, continue... Continue reading the row version identifier in the source logical root and the row version identifier in the target logical root, and verify the row version identifier in the source logical root and the row version identifier in the target logical root. Then read the deletion status identifier in the source logical root and the deletion status identifier in the target logical root, and verify the deletion status identifier in the source logical root and the deletion status identifier in the target logical root. When the row version identifier and the deletion status identifier are the same, the version deletion is confirmed and saved as the version deletion status. When the row version identifier or the deletion status identifier is inconsistent, the version deletion is not confirmed and saved as the version deletion status.

[0045] S5.3: Write the corresponding status of data tables, primary keys, fields, and version deletions into the data content review record in the order of review. In the data content review record, read the corresponding status of data tables, primary keys, fields, and version deletions that indicate a false correspondence, and mark the false correspondences as discrepancies. In the data content review record, read the corresponding status of data tables, primary keys, fields, and version deletions that indicate a true correspondence, and mark the true correspondences as consistency. Then save the discrepancy and consistency marks as the data content consistency status. The data content consistency status is saved as a logical result; after reading the source physical root and the target physical root, the source storage layer identifier, source file fragment identifier, source compressed block identifier, source partition identifier, and source data page identifier are read sequentially from the source physical root. They are sorted according to the subordinate order of source storage layer identifier limiting source file fragment identifier, source file fragment identifier limiting source compressed block identifier, source compressed block identifier limiting source partition identifier, and source partition identifier limiting source data page identifier. The sorted source storage layer identifier, source file fragment identifier, source compressed block identifier, source partition identifier, and source data page identifier are concatenated and saved as the storage ownership path.

[0046] S5.4: Along the source storage layer identifier in the storage home path, verify the corresponding target storage layer identifier in the target physical root; along the source file fragment identifier in the storage home path, verify the corresponding target file fragment identifier in the target physical root; along the source compressed block identifier in the storage home path, verify the corresponding target compressed block identifier in the target physical root; along the source partition identifier in the storage home path, verify the corresponding target partition identifier in the target physical root; along the source data page identifier in the storage home path, verify the corresponding target data page identifier in the target physical root. The verification results of the target storage layer identifier, target file fragment identifier, target compressed block identifier, target partition identifier, and target data page identifier are saved as a storage location relation. The storage layer verification results, file fragment verification results, compressed block verification results, partition verification results, and data page verification results are read from the storage location relation respectively. When the storage layer verification result indicates a true match, the storage layer verification result is marked as a consistent item; when the storage layer verification result indicates a false match, the storage layer verification result is marked as a misaligned item. When the file fragment verification result indicates a true match, the file fragment verification result is stored as a consistent item. When a file segment verification result indicates a mismatch, the file segment verification result is marked as a misaligned item. Similarly, when a compressed block verification result indicates a match, the compressed block verification result is marked as a match; when the compressed block verification result indicates a mismatch, the compressed block verification result is marked as a misaligned item. Likewise, when a partition verification result indicates a match, the partition verification result is marked as a match; when the partition verification result indicates a mismatch, the partition verification result is marked as a misaligned item. Finally, when a data page verification result indicates a match, the data page verification result is marked as a match; when the data page verification result indicates a misalignment, the data page verification result is marked as a match. If the correspondence is not true, the data page verification result is marked as a misalignment item. Only when the storage layer verification result, file fragment verification result, compressed block verification result, partition verification result, and data page verification result all indicate a true correspondence, is the storage location relationship marked as storage ownership consistent. When any one of the verification results of the storage layer verification result, file fragment verification result, compressed block verification result, partition verification result, and data page verification result indicates a false correspondence, the misalignment item of the corresponding level is retained, and the consistency item and the misalignment item are saved as storage ownership consistent state, and the storage ownership consistent state is saved as the physical result.

[0047] S6: Read the logical and physical results, classify and judge them in parallel to form a dual-root result, determine the content difference, storage difference and mixed difference based on the dual-root result, and generate the verification result.

[0048] S6.1: Read the logical results and extract consistent and dissimilar items corresponding to the data table status, primary key status, field status, and version deletion status. Read the physical results and extract consistent and dissimilar items corresponding to the storage layer identifier, file fragment identifier, compressed block identifier, partition identifier, and data page identifier. Arrange the consistent and dissimilar items in the logical results alongside the consistent and dissimilar items in the physical results according to the same primary key identifier and the same field sequence, ensuring that the data content verification results and storage location verification results form a correspondence under the same verification object. When the parallel arrangement results contain consistent items from both the logical and physical results, mark the parallel arrangement results as consistent. When the parallel arrangement results contain dissimilar items from the logical results, mark the parallel arrangement results as content dissimilar. When the parallel arrangement results contain dissimilar items from the physical results, mark the parallel arrangement results as storage dissimilar. Save the parallel arrangement results after classification and marking as a double-root result.

[0049] S6.2: Read the classification markers in the double root result. When all classification markers in the double root result are consistent and there are no differences or misalignments, the data content and storage location between the source database and the target database are determined to be synchronized and consistent. When there are differences but no misalignments in the double root result, it indicates that there are inconsistencies in the data table identifier, primary key identifier, field sequence, field content, row version identifier, or deletion status identifier between the source database and the target database, but the storage location relationship is not misaligned, and the double root result is determined to be a content difference. When there are no differences but misalignments in the double root result, it indicates that the data content between the source database and the target database is consistent, but there are misalignments in the storage layer identifier, file fragment identifier, compressed block identifier, partition identifier, or data page identifier, and the double root result is determined to be a storage difference. When there are both differences and misalignments in the double root result, it indicates that there are both data content inconsistencies and storage location inconsistencies between the source database and the target database, and the double root result is determined to be a mixed difference.

[0050] S6.3: Read the classification results of the dual-root results that have been determined to be synchronous, content-differentiated, storage-differentiated, or mixed-differentiated. Then, read the data table identifier, primary key identifier, field ordinal number, row version identifier, deletion status identifier, data page identifier, partition identifier, file fragment identifier, compressed block identifier, and storage layer identifier from the logical and physical results corresponding to the dual-root results. Organize the read identifiers into a verification detail according to the order of data table identifier, primary key identifier, field ordinal number, row version identifier, deletion status identifier, data page identifier, partition identifier, file fragment identifier, compressed block identifier, and storage layer identifier. Associate the dual-root results with the verification detail and save them together, so that the verification result simultaneously records the difference type and difference location. Finally, save the associated dual-root results and verification detail as the verification result.

[0051] This embodiment also provides a computer device applicable to the database synchronization verification method based on data storage, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the database synchronization verification method based on data storage as proposed in the above embodiment.

[0052] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0053] This embodiment also provides a storage medium on which a computer program is stored. When executed by a processor, the program implements the database synchronization verification method based on data storage as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0054] In summary, this invention, by confirming the comparability of the source and target databases before performing data content comparison, and limiting subsequent verification to data that has been synchronized and is readable, avoids directly judging content when the verification objects are not yet aligned. This reduces misjudgments caused by incomplete synchronization, transaction invisibility, or changes in storage state. The verification results no longer simply reflect whether field values ​​are consistent, but more accurately reflect the synchronization status between the source and target databases. This makes anomaly detection more accurate, difference investigation more focused, and subsequent synchronization compensation and maintenance more targeted, ultimately improving the accuracy, stability, and reliability of database synchronization verification.

[0055] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A database synchronization verification method based on data storage, characterized in that: include, Obtain the database synchronization task, read the storage information of the source database and the target database, and obtain the source storage information and the target storage information; Extract generational elements from the source storage information and the target storage information respectively to obtain source generational information and target generational information, and perform corresponding verification to obtain anchoring results; Based on the anchoring results, determine the verification range under the same storage generation in the source storage information and the target storage information. From the source storage information within the verification range, split the data content elements and storage ownership elements, save the data content elements as the source logical root, and save the storage ownership elements as the source physical root. Using the source logical root and source physical root as verification benchmarks, locate the data content elements and storage ownership elements in the target stored information, and organize them into the target logical root and target physical root. The correspondence between the data content of the source logical root and the target logical root is verified layer by layer to form a logical result. The storage location relationship of the target physical root is checked along the storage ownership path of the source physical root to form a physical result. Read the logical and physical results, classify and judge them side by side to form a dual-root result, determine the content difference, storage difference and mixed difference based on the dual-root result, and generate the verification result.

2. The database synchronization verification method based on data storage as described in claim 1, characterized in that: The database synchronization task includes source database identifier, target database identifier, data table identifier, synchronization batch identifier, synchronization direction identifier, and verification trigger identifier; The source storage information includes source data content elements, source storage ownership elements, and source generation elements; The target-end storage information includes target-end data content elements, target-end storage ownership elements, and target-end generation elements.

3. The database synchronization verification method based on data storage as described in claim 2, characterized in that: The specific steps for obtaining the source generation information and the target generation information are as follows: Read the source-end synchronization batch identifier, source-end table structure version, source-end transaction commit boundary, source-end transaction visibility boundary, source-end data page version, source-end partition version, and source-end storage layer identifier from the source-end storage information, arrange and save them according to the generation record format, and obtain the source-end generation information. Read the target-side synchronization batch identifier, target-side table structure version, target-side transaction commit boundary, target-side transaction visibility boundary, target-side data page version, target-side partition version, and target-side storage layer identifier from the target-side storage information, arrange them according to the generation record format, and save them to obtain the target-side generation information.

4. The database synchronization verification method based on data storage as described in claim 3, characterized in that: The corresponding verification is performed to obtain the anchoring result, specifically as follows: Read the source generation information and target generation information. Using the synchronization batch identifier in the database synchronization task as the verification benchmark, match the source synchronization batch identifier with the target synchronization batch identifier, match the source table structure version with the target table structure version, match the source data page version with the target data page version, match the source partition version with the target partition version, and match the source storage layer identifier with the target storage layer identifier to obtain the generation correspondence record. Read the source transaction commit boundary and the source transaction visibility boundary to determine the source verifiable boundary. Read the target transaction commit boundary and the target transaction visibility boundary to determine the target verifiable boundary. Mark the overlapping part of the source verifiable boundary and the target verifiable boundary as the generation verifiable range. When the generation correspondence record represents the batch correspondence, structure correspondence, data page correspondence, partition correspondence and storage layer correspondence, and the generation verifiable range exists, the source generation information and the target generation information are marked as the same storage generation, and the anchoring result is obtained. When there are uncorresponding entries in the generation-corresponding records and the generation verifiable range does not exist, the source generation information and the target generation information are marked as different storage generations, resulting in an anchoring failure. The anchoring result is either the anchoring success result or the anchoring failure result.

5. The database synchronization verification method based on data storage as described in claim 4, characterized in that: The step of saving data content elements as source-end logical roots and storage attribution elements as source-end physical roots specifically involves: Read the anchoring results. When the anchoring results include anchoring success results, mark the data range in the verifiable range corresponding to the data table identifier as the verification range. Within the verification range, read the source storage information and classify the source primary key identifier, source field content, source field sequence, source row version identifier, and source deletion status identifier into the data content elements. Classify the source data page identifier, source partition identifier, source file fragment identifier, source compressed block identifier, and source storage layer identifier into the storage ownership elements. The data content elements are arranged according to the primary key identifier and the order of the source field, and the arranged data content elements are saved as the source logical root. The storage ownership elements are arranged according to the hierarchical relationship of source data page identifier, source partition identifier, source file fragment identifier, source compressed block identifier, and source storage layer identifier, and the arranged storage ownership elements are saved as the source physical root.

6. The database synchronization verification method based on data storage as described in claim 5, characterized in that: The sorting is divided into target-side logical roots and target-side physical roots, specifically: Read the source-end primary key identifier, source-end field sequence, source-end row version identifier, and source-end deletion status identifier from the source-end logical root to form a data content index; Read the source data page identifier, source partition identifier, source file fragment identifier, source compressed block identifier, and source storage layer identifier from the source physical root to form a storage ownership index; By using data content indexing, the target primary key identifier, target field content, target field ordinal, target row version identifier, and target deletion status identifier are located in the target storage information within the verification range to obtain the target data content elements. Based on the storage ownership index, locate the target data page identifier, target partition identifier, target file fragment identifier, target compressed block identifier, and target storage layer identifier in the target storage information within the verification range to obtain the target storage ownership elements; Organize the target data content elements according to the order in the data content index and save them as the target logical root; Organize the target storage ownership elements according to the hierarchical relationship in the storage ownership index and save them as the target physical root.

7. The database synchronization verification method based on data storage as described in claim 6, characterized in that: The physical result is specifically as follows: Read the source logical root and the target logical root, match the data table identifier in the source logical root with the data table identifier in the target logical root, and obtain the corresponding status of the data table; When the corresponding status of the data table indicates that the data table correspondence is true, the primary key identifier in the source logical root is used as the retrieval basis to find the corresponding primary key identifier in the target logical root to obtain the primary key corresponding status. When the primary key correspondence status indicates that the primary key correspondence is true, the field content in the source logical root and the field content in the target logical root are read sequentially according to the field order, and the field order and field content are checked item by item to obtain the field correspondence status. When the corresponding status of the fields is true, the row version identifier in the source logical root and the row version identifier in the target logical root are reviewed, and the deletion status identifier in the source logical root and the deletion status identifier in the target logical root are also reviewed to obtain the corresponding status of the version deletion. Write the corresponding status of the data table, the corresponding status of the primary key, the corresponding status of the field, and the corresponding status of the version deletion into the data content review record. Mark the non-corresponding items in the data content review record as differences, mark the corresponding items in the data content review record as consistency, and save the difference marks and consistency marks as the data content consistency status. Save the data content consistency status as a logical result. Read the source physical root and the target physical root. Read the storage layer identifier, file fragment identifier, compressed block identifier, partition identifier and data page identifier in the source physical root. Sort them according to the subordinate order of storage layer identifier, file fragment identifier, compressed block identifier, partition identifier and data page identifier, and concatenate the sorted identifiers to save as the storage home path. The storage location relationship is obtained by sequentially checking the storage layer identifier, file fragment identifier, compressed block identifier, partition identifier, and data page identifier in the physical root of the target end along the storage ownership path. Mark the corresponding and non-corresponding items in the storage location relationship as consistent items and misaligned items respectively, and save them as storage ownership consistency status. Save the storage ownership consistency status as physical result.

8. The database synchronization verification method based on data storage as described in claim 7, characterized in that: The specific steps for generating the verification result are as follows: Read the logical and physical results, identify consistent and inconsistent items in the logical results, and identify consistent and misaligned items in the physical results; The consistent and differing items in the logical results are arranged side by side with the consistent and misaligned items in the physical results, and the parallel arrangement results are categorized and marked to form a double root result; When the double root result contains only consistent items, it is determined to be synchronously consistent; when the double root result contains differing items but not misaligned items, it is determined to be content differing; when the double root result does not contain differing items but contains misaligned items, it is determined to be storage differing; when the double root result contains both differing items and misaligned items, it is determined to be mixed differing. Read the data table identifier, primary key identifier, field ordinal number, row version identifier, deletion status identifier, data page identifier, partition identifier, file fragment identifier, compressed block identifier, and storage layer identifier corresponding to the double root result, and organize them into a verification detail; Save the double root result and verification details as the verification result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the database synchronization verification method based on data storage as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the database synchronization verification method based on data storage as described in any one of claims 1 to 8.

Images