Data migration consistency verification method and device, equipment and medium

Abstract

The application discloses a data migration consistency verification method and device, equipment and medium, and belongs to the technical field of computers. The method comprises the following steps: dividing a source data set and a target data set into a plurality of data blocks according to a preset block strategy, and recording data range identification information corresponding to each data block; for each data block, adjusting the field order of each record in the data block according to a preset field order rule, and performing format unification processing on each field value according to a preset field type standardization rule; based on the adjusted and unified data content, determining the hash value of each data block; sorting the source hash set and the target hash set respectively, and comparing bit by bit according to the same sequence position after sorting, and marking the data blocks with inconsistent hash values as difference blocks; for any difference block, comparing the data records indicated by the data range identification information of the difference block piece by piece to determine the data records with differences.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data migration consistency verification method, apparatus, device, and medium. Background Technology

[0002] After data migration (such as database version upgrade, heterogeneous database migration, database sharding, etc.), it is usually necessary to verify the consistency of data between the source database and the target database to ensure the integrity and accuracy of the migrated data.

[0003] Currently, the commonly used verification method is full table verification, which involves exporting the entire data table to be migrated from the source database, calculating the verification value of the entire data table, performing the same calculation on the migrated data table in the target database, and determining whether the migration was successful by comparing whether the verification values ​​of the two ends are consistent.

[0004] However, full table validation can only yield an overall conclusion of consistency or inconsistency. When the validation values ​​are inconsistent, it is impossible to determine which data rows or fields the differences occur in, requiring manual inspection of the entire table row by row, which is extremely inefficient. Summary of the Invention

[0005] This application provides a data migration consistency verification method, apparatus, device, and medium, which can avoid false alarms caused by differences in field order and format, and can accurately locate differing data blocks. The technical solution is as follows: According to one aspect of this application, a data migration consistency verification method is provided, the method comprising: According to the preset partitioning strategy, the source data set of the source database and the target data set of the target database are divided into multiple data blocks, and the data range identification information corresponding to each data block is recorded. For each data block, the field order of each record in each data block is adjusted according to a preset field order rule, and the format of each field value is standardized according to a preset field type standardization rule. Based on the adjusted and standardized data content, the hash value of each data block is determined. The source hash set, composed of the hash values ​​of data blocks in the source data set, and the target hash set, composed of the hash values ​​of data blocks in the target data set, are sorted respectively. Then, the data blocks with inconsistent hash values ​​are marked as difference blocks by comparing them bit by bit according to the same sequence position after sorting. For any discrepancy block, the data records indicated by the data range identifier information of the discrepancy block are compared one by one to determine the data records with discrepancies.

[0006] According to another aspect of this application, a data migration consistency verification apparatus is provided, the apparatus comprising: The partitioning module is used to divide the source data set of the source database and the target data set of the target database into multiple data blocks according to a preset partitioning strategy, and record the data range identification information corresponding to each data block. The first determining module is used to adjust the field order of each record in each data block according to a preset field order rule, and to perform format unification processing on each field value according to a preset field type standardization rule, and to determine the hash value of each data block based on the adjusted and unified data content; The comparison module is used to sort the source hash set composed of the hash values ​​of data blocks in the source data set and the target hash set composed of the hash values ​​of data blocks in the target data set, and compare them bit by bit according to the same sequence position after sorting, and mark the data blocks with inconsistent hash values ​​as difference blocks. The second determining module is used to compare the data records one by one according to the data range identification information of the difference block for any difference block, and determine the data records with differences.

[0007] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the data migration consistency verification method as described above.

[0008] According to another aspect of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the data migration consistency verification method as described above.

[0009] According to another aspect of this application, a computer program product is provided, comprising a computer program executed by a processor to implement the data migration consistency verification method provided in various alternative implementations of the above aspects.

[0010] This application provides a data migration consistency verification scheme. By adjusting the field order of each record within a data block and standardizing the format of each field value, it eliminates data representation inconsistencies caused by different field orders and data type formats between heterogeneous databases, avoiding false alarms during hash comparison. By sorting the hash sets of the source and target databases separately and comparing them bit by bit, it eliminates interference from different physical storage orders of the databases on the verification results, further ensuring comparison accuracy. By recording the data range identifier information corresponding to each data block, and accurately locating the specific data block when hash comparisons are inconsistent, it achieves rapid source tracing and fine-grained location of differences. This scheme can avoid false alarms caused by differences in field order and format, and can accurately locate the differing data blocks. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the implementation environment of a data migration consistency verification method provided in the embodiments of this application; Figure 2 This is a flowchart of a data migration consistency verification method provided according to an embodiment of this application; Figure 3 This is a flowchart illustrating another data migration consistency verification method provided according to an embodiment of this application; Figure 4 This is a flowchart of another data migration consistency verification method provided according to an embodiment of this application; Figure 5 This is a schematic diagram of the data migration consistency verification device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided according to an embodiment of this application.

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0015] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0016] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0017] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the media resources involved in this application were obtained with full authorization.

[0018] Figure 1 This is a schematic diagram illustrating the implementation environment of a data migration consistency verification method provided in an embodiment of this application. See also... Figure 1 The implementation environment specifically includes: terminal device 101 and server 102. Terminal device 101 can be connected to server 102 via wireless network or wired network.

[0019] Terminal device 101 can be at least one of the following: smartphone, smartwatch, desktop computer, laptop, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), and laptop computer. An application can be installed and run on terminal device 101. This application is compatible with various media content platforms, covering scenarios such as music playback, online singing, live streaming, and on-demand video streaming, and can be in the form of a standalone app or mini-program. This application is associated with server 102, which provides background services to terminal device 101.

[0020] Terminal device 101 can refer to one of a plurality of terminal devices. This embodiment uses terminal device 101 as an example. Those skilled in the art will know that the number of the above-mentioned terminal devices can be more or less. For example, there can be several, dozens or hundreds, or more terminal devices. This application embodiment does not limit the number or type of terminal devices.

[0021] Server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of servers can be more or less, and this embodiment does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, server 102 undertakes the main computing work, and terminal device 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal device 101 undertakes the main computing work; or, server 102 and terminal device 101 collaborate on computing using a distributed computing architecture. Server 102 can be connected to terminal device 101 and other terminal devices via a wireless network or a wired network. Optionally, the number of servers can be more or less, and this embodiment does not limit this.

[0022] Figure 2 This is a flowchart of a data migration consistency verification method provided according to an embodiment of this application, such as... Figure 2 As shown, this method is executed by a computer device, which can be... Figure 1 The server 102 shown includes the following steps: Step 201: Divide the source data set of the source database and the target data set of the target database into multiple data blocks according to the preset block division strategy, and record the data range identification information corresponding to each data block.

[0023] In this embodiment of the application, this step divides the source data set in the source database and the target data set in the target database into multiple data blocks according to a preset block division strategy.

[0024] The block partitioning strategy refers to dividing a dataset into multiple logical units according to certain rules. Data blocks are the basic unit of processing; processing data blocks effectively reduces the amount of data processed in a single operation, decreases memory usage, and supports parallel processing.

[0025] Optionally, the preset block segmentation strategy can be to segment according to a preset row number threshold, or to segment according to a preset quantity threshold, or to use a dynamic adaptive segmentation method, that is, to adjust the number of data records contained in a single data block according to the size of the data records.

[0026] Data range identification information refers to information that can determine the range of data records corresponding to a data block. Once a correspondence is established between data blocks and data records, this data range identification information can serve as the basis for subsequent location of discrepancy data.

[0027] For each data block, the data range identifier information corresponding to that data block is recorded. Optionally, when the data set is a database table, the data range identifier information may include, but is not limited to: data block identifier, starting primary key value, ending primary key value, and original field snapshots of each data record within the block.

[0028] For example, consider an order table where the primary key is the order number and it contains 20 million rows. The table is divided into blocks of 1000 rows each. The first block contains records with primary key values ​​from 1 to 1000, the second block contains records with primary key values ​​from 1001 to 2000, and so on. The data range identifiers are recorded as follows: Block ID=1, starting primary key=1, ending primary key=1000; Block ID=2, starting primary key=1001, ending primary key=2000.

[0029] Step 202: For each data block, adjust the field order of each record in each data block according to the preset field order rules, and perform format unification processing on each field value according to the preset field type standardization rules. Based on the adjusted and unified data content, determine the hash value of each data block.

[0030] In this embodiment of the application, for each data block divided in step 201, the following processing is performed: first, the field order of each record in the data block is adjusted according to the preset field order rules; then, the format of each field value is standardized according to the preset field type standardization rules; and finally, the hash value of the data block is calculated based on the adjusted and standardized data content.

[0031] The preset field order rule refers to the rule used to unify the order of fields in each record within a data block. Because the physical storage of data tables in the source and target databases may differ in field order—for example, the source table might have fields arranged as (field A, field B, field C), while the target table might have fields arranged as (field A, field C, field B)—if hash calculations are performed directly on the original data, semantically identical data will generate different hash values ​​due to the different field orders, leading to misjudgments. Therefore, it is necessary to unify the field order of each record before hash calculation.

[0032] Optionally, the global field sorting sequence is determined based on the metadata of the source data tables in the source database. The names and ordinal numbers of each field in the source data tables are obtained and used as a baseline template. Following the field order in this baseline template, the fields of each record in the target data block are reordered, and the same order is applied to each record in the source data block, ensuring that the field order of the data blocks in both databases is completely consistent.

[0033] Predefined field type standardization rules refer to the rules used to unify the format representation of the same semantic fields across different databases. Different databases may have different storage formats and representations for the same data type. For example, for time types, MySQL represents it as 20XX-YY-03 10:30:00, while Oracle may represent it as 03-AUG-20XX 10:30:00 AM; for floating-point numbers, different databases also handle precision in slightly different ways. Without format standardization, the same data content will be represented as different strings in different databases, leading to inconsistent hash values.

[0034] Step 203: Sort the source hash set (composed of the hash values ​​of the data blocks in the source data set) and the target hash set (composed of the hash values ​​of the data blocks in the target data set) respectively, and compare them bit by bit according to the same sequence position after sorting, and mark the data blocks with inconsistent hash values ​​as difference blocks.

[0035] In this embodiment, after calculating the hash values ​​of all data blocks in step 202, this step forms a source hash set from the hash values ​​of all data blocks in the source data set and a target hash set from the hash values ​​of all data blocks in the target data set. Then, the source hash set and the target hash set are sorted to obtain a sorted source hash sequence and a sorted target hash sequence. Finally, according to the same sequence position in the sorted sequence, the hash values ​​at corresponding positions in the source hash sequence and the target hash sequence are compared one by one, and data blocks with inconsistent hash values ​​are marked as difference blocks.

[0036] Sorting the hash set involves independently sorting the hash values ​​of all data blocks in both the source and target databases. Sorting eliminates the interference of differences in the physical storage order of the databases on the hash comparison results. Optionally, a heap sort algorithm can be used to sort the hash set to reduce memory usage.

[0037] Position-by-position comparison refers to comparing the hash values ​​at the same positions in the sorted source hash sequence and the sorted target hash sequence, starting from the first position. For example, if the source hash sequence is [H1,H2, H3, ..., Hn] and the target hash sequence is [K1, K2, K3, ..., Kn], then H1 is compared with K1, H2 with K2, H3 with K3, and so on, until all comparisons are completed.

[0038] When two hash values ​​at a certain position are inconsistent, it indicates that there is a data difference between the data block from the source database and the data block from the target database corresponding to that position, and the data block is marked as a difference block.

[0039] Step 204: For any discrepancy block, compare the data records one by one according to the data range identifier information of the discrepancy block to determine the data records with discrepancies.

[0040] In this embodiment of the application, for any difference block marked in step 203, this step compares the data records within the range indicated by the data range identifier information of the difference block one by one to determine the specific data record with the difference.

[0041] The data range identification information refers to the data range information corresponding to each data block recorded in step 201, including but not limited to the data block identifier, starting primary key value, ending primary key value, and original field snapshots of each data record within the block. After the difference blocks are marked in step 203, the location operation only needs to be performed on these difference blocks, without processing all data blocks, thus improving processing efficiency.

[0042] Optionally, the data range identifier information of the data block marked as a difference block is first obtained, and the start and end primary key values ​​corresponding to the data block are read from Redis based on the data range identifier information. Then, based on the primary key value range, the original data records within that range are extracted from the source and target databases respectively. Since the data blocks are divided according to a preset block partitioning strategy, the number of data records contained in each data block is limited (e.g., 1000 rows), so the extraction operation only involves a small number of data records.

[0043] After obtaining the data records corresponding to the difference blocks, each data record is compared line by line. This line-by-line comparison means comparing the content of each data record in the source and target databases to ensure complete consistency. The comparison can include the values ​​of each field in the data record. When a data record is found to be inconsistent between the source and target databases, that record is identified as the data record with discrepancies.

[0044] Information from difference data records can be logged and output, such as the primary key value, difference row number, difference field name, field value in the source database, and field value in the target database. This information can be organized into a structured difference report for data repair or rollback operations.

[0045] This application provides a data migration consistency verification scheme. By adjusting the field order of each record within a data block and standardizing the format of each field value, it eliminates data representation inconsistencies caused by different field orders and data type formats between heterogeneous databases, avoiding false alarms during hash comparison. By sorting the hash sets of the source and target databases separately and comparing them bit by bit, it eliminates interference from different physical storage orders of the databases on the verification results, further ensuring comparison accuracy. By recording the data range identifier information corresponding to each data block, and accurately locating the specific data block when hash comparisons are inconsistent, it achieves rapid source tracing and fine-grained location of differences. This scheme can avoid false alarms caused by differences in field order and format, and can accurately locate the differing data blocks.

[0046] The above Figure 2 The diagram shows the main flow of a data migration consistency verification method provided in this application. The following is a further explanation of this data migration consistency verification method. Figure 3 This is a flowchart illustrating another data migration consistency verification method provided according to an embodiment of this application. The method is executed by a computer device, such as... Figure 3 As shown, the method includes: Step 301: Divide the source data set of the source database and the target data set of the target database into multiple data blocks according to the preset block division strategy, and record the data range identification information corresponding to each data block.

[0047] In this embodiment of the application, the computer device is equipped with a migration verification service. The migration verification service can read data from the source data table of the source database and the target data table of the target database according to a preset block strategy to obtain a source data set and a target data set. Then, the source data set and the target data set are each divided into multiple data blocks, and the data range identification information corresponding to each data block is recorded.

[0048] The block partitioning strategy refers to the method of dividing a data set into multiple logical units according to certain rules. The data block is the basic unit in this scheme. Processing data blocks effectively reduces the amount of data in a single computation, reduces memory usage, and supports parallel processing.

[0049] Data range identification information refers to information that can uniquely identify the range of data records corresponding to a data block. Once a correspondence is established between a data block and a data record, this data range identification information can be used to locate the data record. In this embodiment, the data range identification information can be stored in Redis, and it may include the data block ID, the starting primary key value, the ending primary key value, and a snapshot of the original fields of each data record within the data block.

[0050] Optionally, this application provides three preset segmentation strategies as examples.

[0051] The first method is to divide the data into blocks with a fixed number of rows.

[0052] In this partitioning strategy, the source and target datasets are divided into multiple data blocks according to a preset row count threshold. Each data block contains the number of data records corresponding to the preset row count threshold, and the last data block may contain a number of records less than or equal to this threshold.

[0053] For example, taking an order table as an example, the row count threshold is 1000 rows. The source dataset is divided into 20,000 data blocks. The first data block contains data records with primary key values ​​from 1 to 1000, the second data block contains data records with primary key values ​​from 1001 to 2000, and so on, with the 20,000th data block containing data records with primary key values ​​from 19999001 to 20000000. The target dataset is divided into blocks according to the same row count threshold and partitioning method.

[0054] The second method is to divide the data into blocks with a fixed amount of data.

[0055] In this data partitioning strategy, the source and target datasets are divided into multiple data blocks according to a preset data volume threshold. A data block is formed when the accumulated data volume reaches the preset threshold, and the data volume of the last data block can be less than or equal to the threshold.

[0056] For example, taking a data volume threshold of 10MB as an example, the migration verification service reads data line by line from the source dataset. When the cumulative data volume read reaches 10MB, the read data lines are divided into data blocks, and then the next data block is read. The target dataset is divided in the same way.

[0057] This approach is suitable for data tables where the size of a single row of data varies greatly, such as tables containing fields of type TEXT or BLOB. When the size of a single row of data varies significantly, dividing the data into blocks with a fixed number of rows may result in a large difference in the amount of data in different blocks, affecting the efficiency and balance of subsequent hash calculations. Dividing the data into blocks with a fixed amount of data can effectively solve this problem.

[0058] The third type is dynamic adaptive block partitioning.

[0059] In this block partitioning strategy, the amount of data in each data record in the source data set and the target data set is detected. If the amount of data in a single data record in the current data block exceeds a preset threshold, the number of records contained in the current data block is reduced.

[0060] For example, taking a single row data size threshold of 1KB as an example. During the block partitioning process, the migration verification service reads data records row by row. If it finds that the size of a data record to be added to the current data block exceeds 1KB, the data block is closed in advance, so that the large data record is in a separate block or only in a block with a few other records, avoiding the inclusion of too many large fields in a single data block. This method is used to optimize the partitioning effect of large field tables such as TEXT and BLOB, and can effectively avoid memory overflow or performance degradation in data block hash calculations caused by excessively large single rows of data.

[0061] For each data block, the migration verification service records the data range identifier information corresponding to that data block after completing the block partitioning. The data range identifier information includes, but is not limited to: data block identifier, starting primary key value, ending primary key value, and original field snapshots of each data record within the block.

[0062] For example, for the first data block of the order table, the data range identification information of the record is: Block ID=1, starting primary key=1, ending primary key=1000, and the records in the block are 1000 complete data records with order numbers from 1 to 1000.

[0063] Optionally, the aforementioned data range identification information is stored in Redis. Redis uses a key-value pair structure, with the data block identifier as the key and the start primary key, end primary key, and original field snapshot as the values. In subsequent difference localization steps, the migration verification service can directly read the corresponding data range information from Redis based on the data block identifier, without needing to rescan the source or target database, thus significantly improving localization efficiency.

[0064] Step 302: For each data block, adjust the field order of each record in each data block according to the preset field order rules.

[0065] In this embodiment of the application, for each data block divided in step 301, the migration verification service adjusts the field order of each record in the data block according to the preset field order rules.

[0066] The preset field order rule refers to the rule used to unify the order of fields in each record within a data block. Because the physical storage of data tables in the source and target databases may differ in field order—for example, the source database table might have fields arranged as (order number, order amount, order time, customer name), while the target database table might have fields arranged as (order number, customer name, order amount, order time)—if hash calculations are performed directly on the original data, semantically identical data will generate different hash values ​​due to the different field orders, leading to misjudgments. Therefore, it is necessary to unify the field order of each record before hash calculation.

[0067] In some embodiments, the field order of each record within each data block is adjusted according to a preset field order rule, including the following sub-steps.

[0068] Sub-step 302A: Determine the global field sorting sequence based on the metadata of the source data table in the source database.

[0069] The migration verification service reads metadata information from the source database, extracts all field names and field numbers from the source database, and uses them as a baseline template. Field numbers refer to the physical storage order of each field in the source database.

[0070] For example, if the metadata of the source order table displays the following field order: field number 1 is order_id, field number 2 is amount, field number 3 is create_time, and field number 4 is customer_name, then the base template is: [(1, order_id), (2, amount), (3, create_time), (4, customer_name)], and the global field sorting sequence is determined as: [order_id, amount, create_time, customer_name].

[0071] Optionally, when determining the global field sorting sequence, it is also necessary to handle the matching of field names between the source and target tables. Accordingly, firstly, the field names and field numbers from the source table are extracted as a baseline template. Then, the migration verification service performs a case-insensitive match between the fields in the target table of the target database and the baseline template, based on the field names. For example, if the field name in the source table is `order_id` and the field name in the target table is `ORDER_ID`, and they match successfully, the `ORDER_ID` field in the target table is mapped to the position corresponding to `order_id` in the global field sorting sequence. Finally, the successfully matched fields are mapped to their corresponding positions in the global field sorting sequence according to the order indicated by the field numbers in the baseline template. If a field exists in the target table that is not in the baseline template, or a field exists in the baseline template that is not in the target table, then the match fails, and the unmatched fields are marked as non-matching fields. These non-matching fields do not participate in subsequent field order adjustments and hash value calculations.

[0072] For example, the base template fields in the source order table are [order_id, amount, create_time, customer_name], while the actual fields in the target order table are [ORDER_ID, CUSTOMER_NAME, AMOUNT, CREATE_TIME, NOTE]. After case-insensitive field name matching, ORDER_ID, CUSTOMER_NAME, AMOUNT, and CREATE_TIME all match successfully and are mapped to their corresponding positions. The additional NOTE field in the target order table does not match successfully and is marked as a non-matching field, thus not participating in further processing.

[0073] Sub-step 302B: Sort each field of each record in each data block according to the global field sorting sequence.

[0074] For each data record within each data block of the source dataset, rearrange the field order according to the global field sorting sequence. Similarly, for each data record within each data block of the target dataset, rearrange the field order according to the global field sorting sequence.

[0075] For example, the migration verification service extracts the values ​​of the corresponding fields from each record sequentially according to the order of the fields in the global field sorting sequence, forming a new ordered record. After sorting, the field order of the source data record and the target data record is completely consistent, both being [order_id, amount, create_time, customer_name].

[0076] In some embodiments, before adjusting the field order, the migration verification service can also filter out database hidden fields and internal fields other than auto-incrementing primary keys, preventing these fields from participating in field order adjustment and hash value calculation. Hidden fields refer to internal fields automatically generated by the database and invisible to users, such as ROWID in Oracle. Internal fields other than auto-incrementing primary keys are auxiliary fields used within the database to manage data storage; these fields are automatically maintained by the database and are not part of the business data. Filtering out these fields, preventing them from participating in field order adjustment and hash value calculation, can avoid misjudgments in hash comparisons caused by differences in system fields.

[0077] Step 303: Standardize the format of each field value according to the preset field type standardization rules.

[0078] In this embodiment of the application, for each data block after the field order adjustment is completed, the migration verification service performs format unification processing on the values ​​of each field according to the preset field type standardization rules.

[0079] The pre-defined field type standardization rules refer to the rules used to unify the format representation of fields with the same semantics in different databases. Different databases may have different storage formats and representations for the same data type.

[0080] For example, for time data types, MySQL represents it as 20XX-YY-03 10:30:00, while Oracle might represent it as 03-AUG-20XX 10:30:00 AM; for floating-point numbers, different databases handle precision differently. Without standardized formatting, the same data will be represented as different strings in different databases, leading to inconsistent hash values.

[0081] This application exemplarily illustrates five scenarios for performing format standardization processing.

[0082] The first type is where the field value is a numeric type.

[0083] Numeric types include integer, floating-point, and high-precision decimal (Decimal) types. If the field value is a numeric type, the field value will be uniformly converted to a decimal string representation, leading zeros will be removed, and the preset number of decimal places will be retained.

[0084] For integer values, they are directly converted to decimal strings, and leading zeros are removed. For example, the integer value 000123 is converted to 123.

[0085] For floating-point numbers or high-precision decimals, a preset number of decimal places is uniformly retained. For example, if the preset number of decimal places is set to 10, the floating-point value 123.456 is converted to 123.4560000000; the floating-point value 123.45678912345 is converted to 123.4567891235 (rounded to 10 decimal places). The converted string must not be represented in scientific notation, ensuring that the output is in a regular decimal format.

[0086] The second type is that the field value is of a time type.

[0087] Time types include the DateTime, Timestamp and Date types. If the field value is of a time type, the field value is uniformly converted into a date-time string in a preset format.

[0088] For example, the globally unified date-time format is yyyy-MM-dd HH:mm:ss.SSS, with 3 decimal places reserved for the millisecond part. The time value 20XX-YY-03 10:30:00 in the source database is converted to 20XX-YY-03 10:30:00.000; the time value 03 / Aug / 20XX 10:30:00 AM of the same semantics in the target database is also converted to 20XX-YY-03 10:30:00.000. After this standardization process, time values with the same semantics have completely consistent string representations in different databases.

[0089] For null time values, a preset placeholder is uniformly filled. For example, a NULL value is filled with NULL_TIME.

[0090] The third type is the string type.

[0091] String types include the Varchar and Text types. If the field value is of a string type, leading and trailing whitespace characters of the field value are removed. The whitespace characters include spaces, tabs and newlines.

[0092] For example, the string value " Zhang San " becomes "Zhang San" after processing, with the whitespace characters removed; the string value "\tLi Si\n" becomes "Li Si" after processing.

[0093] For empty strings or NULL values, they are uniformly replaced with a preset placeholder. For example, both the empty string "" and the NULL value are replaced with "NULL_STR", thereby effectively distinguishing empty strings from NULL values.

[0094] The fourth type is that the field value is of a JSON type.

[0095] If the field value is of a JSON type, the key names in the JSON object of the field value are sorted in alphabetical order, and redundant whitespace characters are removed.

[0096] For example, a JSON object originally represented as {b:2, a:1, c:3} can be normalized to {a:1,b:2,c:3}. Alternatively, a JSON object originally containing line breaks and indentation can also be normalized to produce a compact format with keys sorted alphabetically and no extra whitespace characters.

[0097] By sorting the keys of a JSON object and removing extra whitespace characters, it is possible to ensure that JSON objects with the same semantics but different key orders or formatting will output completely identical strings after processing, thereby avoiding inconsistent hash values ​​caused by differences in JSON format.

[0098] The fifth type is a binary large field type.

[0099] Binary large field types include the Blob type. If a field value is of binary large field type, the binary data of the field value will be uniformly converted into a Base64 encoded string. The encoding format is globally unique, ensuring that the binary data in the source database and the target database have the same Base64 encoded representation under the same semantics.

[0100] For example, binary data stored in a Blob field is Base64 encoded to obtain the string SGVsbG8gV29ybGQ=. The same binary data at the source and destination ends will result in the same Base64 encoded string after conversion.

[0101] Step 304: Based on the adjusted and unified data content, determine the hash value of each data block.

[0102] In this embodiment of the application, this step is based on the data content after the field order adjustment in step 302 and the format unification processing in step 303, and the migration verification service determines the hash value of each data block.

[0103] Optionally, this step includes the following sub-steps.

[0104] Sub-step 304A: For each data block, firstly, concatenate the unified field values ​​of each record within the data block according to the adjusted field order to obtain the record string for each record. During the concatenation process, a preset unique delimiter is inserted between adjacent field values. The unique delimiter is a specific character or string that will not appear in any field value, used to prevent the contents of different fields from being concatenated together, thus avoiding hash distortion. For example, an invisible character can be used as the delimiter.

[0105] For a single record within a data block, assuming the field order is [order_id, amount, create_time, customer_name], and the standardized values ​​are 1001, 999.9900000000, 20XX-YY-03 10:30:00.000, and Zhang San, then after inserting delimiters between adjacent field values, the concatenated result of this record will be a combination of the aforementioned field values ​​and delimiters.

[0106] For a data block containing multiple records, after concatenating each record, the record strings of all records within the data block are further concatenated in line order to obtain the complete original text block (i.e., the block string). During line-by-line concatenation, line separators can be inserted between adjacent records to distinguish between different records.

[0107] Sub-step 304B: Perform hash calculation on the original text block of each data block to obtain the hash value of each data block.

[0108] Hash calculations can employ the SHA-256 algorithm to ensure the uniqueness and collision resistance of hash values. In an alternative embodiment, the BLAKE3 algorithm can also be used, which is approximately three times faster than SHA-256 on a quad-core CPU.

[0109] For example, for the first data block in the order table, the block string of the data block is calculated using SHA-256 to obtain the corresponding hash value, which serves as the unique digital fingerprint of the data block for subsequent comparison and verification.

[0110] Optionally, after the hash values ​​of all data blocks have been calculated, the migration verification service can cache the hash results in a Kafka queue for consumption in subsequent sorting and comparison steps.

[0111] Step 305: Sort the source hash set (composed of the hash values ​​of the data blocks in the source data set) and the target hash set (composed of the hash values ​​of the data blocks in the target data set) respectively, and compare them bit by bit according to the same sequence position after sorting, and mark the data blocks with inconsistent hash values ​​as difference blocks.

[0112] In this embodiment, after calculating the hash values ​​of all data blocks in step 304, the migration verification service forms a source hash set from the hash values ​​of all data blocks in the source data set and a target hash set from the hash values ​​of all data blocks in the target data set. Then, the source hash set and the target hash set are sorted to obtain a sorted source hash sequence and a sorted target hash sequence. Finally, according to the same sequence position after sorting, the hash values ​​at corresponding positions in the source hash sequence and the target hash sequence are compared one by one, and data blocks with inconsistent hash values ​​are marked as difference blocks.

[0113] Sorting the hash set refers to independently sorting the hash values ​​of all data blocks in both the source and target databases. This sorting eliminates the interference of differences in the physical storage order of the databases on the hash comparison results. Because the source and target databases may have different physical storage orders for the same table—for example, the source database might store data in ascending order by primary key, while the target database might have a different storage order due to different database configurations or import methods—if the comparison is performed directly according to the original order of the data blocks, even if the data content in both databases is completely identical, the different order will lead to a large number of hash value mismatches and false alarms. By sorting the hash values ​​before comparison, these order differences can be effectively eliminated.

[0114] Optionally, the migration verification service can use a heap sort algorithm to sort the hash set to reduce memory usage. For a scenario with 20 million rows of data, divided into blocks of 1,000 rows each, there are 20,000 hash values ​​involved in the sorting, resulting in a memory footprint of approximately several MB, far lower than the storage requirements for the entire dataset.

[0115] Each hash value in the sorted hash sequence remains associated with its corresponding data block identifier. For example, if the source hash set before sorting is [(Block_ID=1, Hash=H1), (Block_ID=2, Hash=H2),...], the sorted source hash sequence is [H1, H2, H3, ..., Hn], where each hash value can still be traced back to its corresponding Block_ID through the association.

[0116] Position-by-position comparison refers to comparing the hash values ​​at the same positions in the sorted source hash sequence and the sorted target hash sequence, starting from the first position. For example, if the source hash sequence is sorted as [HS1, HS2, HS3, ..., HSn] and the target hash sequence is sorted as [HT1, HT2, HT3, ..., HTn], then HS1 is compared with HT1, HS2 with HT2, HS3 with HT3, and so on, until all comparisons are completed.

[0117] When two hash values ​​at a certain position are inconsistent, it indicates that there is a data difference between the data block from the source database and the data block from the target database corresponding to that position, and the data block is marked as a difference block. For positions with consistent hash values, it means that the data block from the source database and the data block from the target database corresponding to that position are completely identical in data content (assuming the probability of hash collision is negligible), and no further inspection of the data within that data block is required.

[0118] For example, if the source hash value at the 5th position after sorting is inconsistent with the target hash value, then the data block from the source database and the data block from the target database corresponding to the 5th position are marked as difference blocks. If the source hash value at the 10th position after sorting is the same as the target hash value, then the data block corresponding to that position passes verification and requires no further processing.

[0119] Step 306: For any discrepancy block, compare the data records one by one according to the data range identifier information of the discrepancy block to determine the data records with discrepancies.

[0120] In this embodiment of the application, for any difference block marked in step 305, the migration verification service compares the data records within the range indicated by the data range identifier information of the difference block, and determines the specific data records with differences.

[0121] The data range identification information refers to the data range information corresponding to each data block recorded in step 301, including but not limited to the data block identifier, starting primary key value, ending primary key value, and original field snapshots of each data record within the block. After the difference blocks are marked in step 305, the location operation only needs to be performed on these difference blocks, without processing all data blocks.

[0122] In this embodiment, the migration verification service automatically selects different positioning strategies based on the number of difference blocks. First, the number of difference blocks is obtained.

[0123] Then, in the first scenario, if the number is less than a preset threshold (e.g., 10), it indicates that the number of blocks with inconsistent hash comparisons is small, belonging to a scenario with only minor differences. In this case, a concurrent backtracking algorithm is used to locate the discrepancies.

[0124] Accordingly, the migration verification service batch-obtains the identification information of all difference blocks, and reads the start and end primary key values ​​corresponding to each difference block from Redis based on the identification information of each difference block. Then, it starts multi-threading to read the original data records within the primary key range corresponding to each difference block from the source and target databases in parallel. That is, it concurrently reads each data record indicated by the data range identification information corresponding to the difference block. Since the number of difference blocks is small, and each difference block contains a limited number of data records (e.g., 1000 rows), concurrent reading can fully utilize multi-core CPU resources to quickly complete data extraction. Finally, it performs a line-by-line comparison to identify data records with differences. Accordingly, for each extracted data record, it re-executes the field sorting and concatenation operation line by line, and then compares the string content field by field. When a data record is found to be inconsistent between the source and target databases, it marks the row number, the name of the abnormal field, the field value in the source database, and the field value in the target database. This method is fast and has low IO overhead in scenarios with a small number of differences.

[0125] Then, in the second scenario, if the number is greater than or equal to a preset threshold (e.g., 10), it indicates a large area of ​​mismatch in the hash sequence and a large number of differing blocks. In this case, a binary search algorithm is used to locate the discrepancies, avoiding excessive database queries caused by processing a large number of differing blocks one by one.

[0126] Accordingly, the migration verification service first determines the contiguous block interval containing all differing blocks. This contiguous interval is then bisected, dividing the large interval into two sub-intervals. The overall hash value (i.e., the combined hash value of all data blocks within each sub-interval) is calculated and compared with the overall hash value of the corresponding sub-interval on the target end to determine the sub-interval containing the abnormal data. Then, the sub-intervals containing the abnormal data are recursively bisected until the smallest granularity of the abnormal data block is located. In simpler terms, the contiguous block interval containing the differing blocks is recursively bisected; after determining the differing sub-intervals, each data record within the differing sub-interval is compared to identify the records containing the differences.

[0127] For example, suppose the discrepancy blocks are continuously distributed within the range of Block_ID=100 to Block_ID=199. This range is split into two sub-ranges: [100, 149] and [150, 199]. The overall hash value of the sub-range [100, 149] is calculated in both the source and target databases. If they match, it means the anomalies are all located in the [150, 199] range. The [150, 199] range is then further split into two sub-ranges until the specific anomalous data block is located. This method significantly reduces the number of database queries, avoids traversing the entire range, and effectively reduces IO pressure in high-discrepancy scenarios.

[0128] It should be noted that regardless of the location strategy used, the final location will be the specific data block with discrepancies. Subsequently, the migration verification service will compare the content of each data record within the located data block to determine the specific data record with discrepancies.

[0129] Item-by-item comparison refers to comparing each data record individually with the fields in the source and target databases to ensure complete consistency. If a field value in a data record is found to be inconsistent between the source and target databases, that record is identified as having discrepancies, and its primary key value, row number, field name, field value in the source database, and field value in the target database are recorded.

[0130] Step 307: Output a structured difference report.

[0131] In this embodiment, after locating the difference data in step 306, the migration verification service outputs a structured difference report. The structured difference report is a collection of difference information organized according to a preset data format, used to visually display all the details of the differences discovered during the verification process. This report supports exporting in CSV or JSON format and can be directly used for generating data repair or rollback scripts.

[0132] Optionally, the structured difference report includes at least one of the following: difference block identifier, primary key range of difference data records, difference row number, difference field name, field value in the source database, and field value in the target database.

[0133] The difference block identifier is used to uniquely identify data blocks that have differences, for example, Block_ID=5.

[0134] The primary key range of the difference data record is used to indicate the primary key range of the difference data record in the source table and the target table. For example, the starting primary key = 5001 and the ending primary key = 6000.

[0135] The difference row number is used to indicate the specific row number of the difference data record within the data block, such as row 3.

[0136] The difference field name is used to indicate the specific field name where there is a difference, such as the amount field.

[0137] Field value in the source database: Used to display the actual value of the field from the source, such as 999.9900000000.

[0138] The field value in the target database is used to display the actual value of that field on the target side, such as 999.99.

[0139] For example, one output difference detail might be: Difference Block ID=5, Primary Key Range=5001-6000, Row Number=3, Field Name=amount, Source Value=999.9900000000, Target Value=999.99. Users can quickly locate specific data inconsistencies based on this structured difference report, enabling them to perform data repair or rollback operations without the need for manual review of the entire table.

[0140] The following is an appendix Figure 4 Let me further explain this plan. Figure 4 This is a flowchart of another data migration consistency verification method provided according to an embodiment of this application. For example... Figure 4 As shown, the process includes the following steps: 401. Reading data in blocks: Read data from the source database and the target database according to a preset block-based strategy, dividing the source and target data sets into multiple data blocks, and synchronously storing the block ID and corresponding primary key range metadata of each data block to Redis; 402. Field sorting and concatenation: For each data block, organize the field order of each record according to the metadata baseline sequence in the source database, perform unified formatting on special data types, and insert unique delimiters between fields for concatenation; 403. Calculating the hash value of a single block: Calculate the SHA-256 hash value of the complete text content after concatenation of each data block; 404. Sorting the full hash set: Construct hash sets from the hash values ​​of all data blocks in the source and target databases, and sort the two sets using a low-memory optimized sorting algorithm; 405. Comparing the ordered double-ended hash sequence item by item: The sorted source hash sequence and target hash sequence are compared item by item according to the same sequence position; 406. Consistency determination: If the hash values ​​of all corresponding positions are consistent, the data verification is determined to be successful; 407. Output verification success report; If there are data blocks with inconsistent hash values, the data block is marked as a difference block; 408. Extract the block ID of all difference blocks; 409. Adaptive difference fragment location: The location strategy is automatically selected according to the number of difference blocks. If the number of difference blocks is less than the preset threshold, the concurrent backtracking method is used to compare and locate row by row. If the number of difference blocks is greater than or equal to the preset threshold, the binary recursive method is used to split and locate, relying on the mapping relationship between the block ID and the primary key range stored in Redis for tracing; 410. Output structured difference detail report: Output a row-level and field-level structured difference detail report containing the difference block ID, primary key range, difference row number, difference field name, source field value and target field value.

[0141] To verify the technical effectiveness of the technical solution provided in this application, the following comparative tests were also conducted.

[0142] The test environment consisted of two 4-core, 16GB cloud servers, with the source database being MySQL 5.7 and the target database being MySQL 8.0. The test dataset was an order table containing 20 million rows of data, including 18 fields of types such as timestamps, JSON, and floating-point numbers. Comparisons included: traditional full-table MD5 checksum, random sampling with 1% alignment, and the solution provided in this embodiment (with a block size of 1000 rows). The main metrics of each solution are compared below: Traditional full-table MD5 verification requires transmitting all data, taking a total of 47 minutes, with a network transmission volume of approximately 7.2GB and peak memory usage of approximately 3.2GB. While this method has a 0% false positive rate, it cannot pinpoint discrepancies or identify special types of errors.

[0143] With a random sampling method of 1%, the total verification time is 2.3 minutes, the network transmission volume is approximately 72MB, and the peak memory usage is approximately 200MB. However, since only 1% of the data is covered, there is a detection blind spot, the accuracy of difference localization cannot be guaranteed, and the out-of-order data may lead to missed detections. It also cannot identify special types of errors.

[0144] This solution, while ensuring full coverage, takes a total verification time of 3.8 minutes, approximately one-twelfth the time required for full MD5 verification; network transmission volume is approximately 28MB, a reduction of about 99.6% compared to full data transmission; peak memory usage is approximately 480MB. This solution eliminates the interference of physical storage order on verification results through sorting operations, resulting in a 0% false alarm rate; through field sorting and concatenation and type standardization, it can effectively identify special types of errors caused by differences in field order and data type format; simultaneously, this solution provides precise difference location capabilities at both the row and field levels.

[0145] This application provides a data migration consistency verification scheme. By adjusting the field order of each record within a data block and standardizing the format of each field value, it eliminates data representation inconsistencies caused by different field orders and data type formats between heterogeneous databases, avoiding false alarms during hash comparison. By sorting the hash sets of the source and target databases separately and comparing them bit by bit, it eliminates interference from different physical storage orders of the databases on the verification results, further ensuring comparison accuracy. By recording the data range identifier information corresponding to each data block, and accurately locating the specific data block when hash comparisons are inconsistent, it achieves rapid source tracing and fine-grained location of differences. This scheme can avoid false alarms caused by differences in field order and format, and can accurately locate the differing data blocks.

[0146] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0147] It should be noted that the order of the method steps provided in the embodiments of this application can be appropriately adjusted, and the steps can also be added or removed as appropriate. Any method variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0148] Figure 5 This is a schematic diagram of the data migration consistency verification device provided according to an embodiment of this application. Figure 5 As shown, the device includes: a division module 501, a first determination module 502, a comparison module 503, and a second determination module 504.

[0149] The partitioning module 501 is used to divide the source data set of the source database and the target data set of the target database into multiple data blocks according to a preset partitioning strategy, and record the data range identification information corresponding to each data block. The first determining module 502 is used to adjust the field order of each record in each data block according to a preset field order rule, and to perform format unification processing on each field value according to a preset field type standardization rule, and to determine the hash value of each data block based on the adjusted and unified data content. The comparison module 503 is used to sort the source hash set composed of the hash values ​​of the data blocks in the source data set and the target hash set composed of the hash values ​​of the data blocks in the target data set, and compare them bit by bit according to the same sequence position after sorting, and mark the data blocks with inconsistent hash values ​​as difference blocks. The second determining module 504 is used to compare the data records one by one according to the data range identification information of any difference block to determine the data records with differences.

[0150] In some embodiments, the partitioning module 501 is used to partition the source data set and the target data set into multiple data blocks according to a preset row number threshold; or, partition the source data set and the target data set into multiple data blocks according to a preset data volume threshold; or, detect the data volume of each data record in the source data set and the target data set, and if the data volume of a single data record in the current data block exceeds the preset threshold, reduce the number of records contained in the current data block.

[0151] In some embodiments, the first determining module 502 is used to determine a global field sorting sequence based on the metadata of the source data table of the source database; and to sort each field of each record in each data block according to the global field sorting sequence.

[0152] In some embodiments, the first determining module 502 is used to extract the field names and field numbers from the source data table as a baseline template; to perform case-insensitive matching between the fields in the target data table of the target database and the baseline template according to the field names; to map the successfully matched fields to the corresponding positions in the global field sorting sequence according to the order indicated by the field numbers in the baseline template; and to mark the fields that fail to match as unmatched fields.

[0153] In some embodiments, the first determining module 502 is further configured to filter out internal fields other than hidden fields and auto-incrementing primary keys, so that hidden fields and internal fields do not participate in field order adjustment and hash value calculation.

[0154] In some embodiments, the first determining module 502 is further configured to implement any of the following: If the field value is a numeric type, the field value will be uniformly converted to a decimal string representation, leading zeros will be removed, and the preset number of decimal places will be retained; If the field value is of type time, the field value will be uniformly converted into a date and time string in a preset format; If the field value is a string, remove leading and trailing whitespace characters from the field value and replace empty strings or NULL values ​​with preset placeholders; If the field value is of type JSON, then the keys in the JSON object of the field value are sorted alphabetically and extra whitespace characters are removed; If the field value is a binary large field type, then the binary data of the field value will be uniformly converted into a Base64 encoded string.

[0155] In some embodiments, the first determining module 502 is used to concatenate the unified field values ​​sequentially according to the adjusted field order, and insert a preset unique separator between adjacent field values ​​to obtain the original text block of each data block; and to perform hash calculation on the original text block of each data block to obtain the hash value of each data block.

[0156] In some embodiments, the second determining module 504 is used to obtain the number of difference blocks; if the number is less than a preset threshold, it concurrently reads each data record indicated by the data range identifier information corresponding to the difference block, performs a comparison of each content, and determines the data record with difference; if the number is greater than or equal to the preset threshold, it recursively divides the continuous block interval where the difference block is located into two parts, determines the difference sub-intervals, and then performs a comparison of each content of the data records in the difference sub-intervals to determine the data record with difference.

[0157] In some embodiments, the apparatus further includes: The output module is used to output a structured difference report, which includes at least one of the following: difference block identifier, primary key range of difference data records, difference row number, difference field name, field value in the source database, and field value in the target database.

[0158] This application provides a data migration consistency verification device. By adjusting the field order of each record within a data block and standardizing the format of each field value, it eliminates data representation inconsistencies caused by different field orders and data type formats between heterogeneous databases, thus avoiding false alarms during hash comparison. By sorting the hash sets of the source and target databases separately and comparing them bit by bit, it eliminates interference from different physical storage orders of the databases on the verification results, further ensuring comparison accuracy. By recording the data range identifier information corresponding to each data block, and accurately locating the specific data block when hash comparisons are inconsistent, it achieves rapid source tracing and fine-grained location of discrepancies. This solution can avoid false alarms caused by differences in field order and format, and can accurately locate discrepant data blocks.

[0159] It should be noted that the data migration consistency verification device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data migration consistency verification device and the data migration consistency verification method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0160] Embodiments of this application also provide a computer device, which includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the data migration consistency verification method provided in the above-described method embodiments.

[0161] Figure 6This is a schematic diagram of the structure of a computer device provided according to an embodiment of this application.

[0162] Computer device 600 includes a central processing unit (CPU) 601, a system memory 604 including random access memory (RAM) 602 and read-only memory (ROM) 603, and a system bus 605 connecting the system memory 604 and the CPU 601. Computer device 600 also includes a basic input / output system (I / O system) 606 that facilitates information transfer between various devices within the computer device, and a mass storage device 607 for storing the operating system 613, application programs 614, and other program modules 615.

[0163] The basic input / output system 606 includes a display 608 for displaying information and an input device 609 for user input, such as a mouse or keyboard. Both the display 608 and the input device 609 are connected to the central processing unit 601 via an input / output controller 610 connected to the system bus 605. The basic input / output system 606 may also include the input / output controller 610 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 610 also provides output to a display screen, printer, or other types of output devices.

[0164] Mass storage device 607 is connected to central processing unit 601 via a mass storage controller (not shown) connected to system bus 605. Mass storage device 607 and its associated computer-readable storage media provide non-volatile storage for computer device 600. That is, mass storage device 607 may include computer-readable storage media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drive.

[0165] Without loss of generality, computer-readable storage media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technique for storing information such as computer-readable storage instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically-erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 604 and mass storage device 607 described above can be collectively referred to as memory.

[0166] The memory stores one or more programs, which are configured to be executed by one or more central processing units 601. The one or more programs contain instructions for implementing the above method embodiments, and the central processing unit 601 executes the one or more programs to implement the methods provided by the various method embodiments described above.

[0167] According to various embodiments of this application, computer device 600 can also be connected to and operated on a remote computer device on a network, such as the Internet. That is, computer device 600 can be connected to network 612 through network interface unit 611 connected to system bus 605, or network interface unit 611 can be used to connect to other types of networks or remote computer device systems (not shown).

[0168] The memory also includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0169] This application also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the data migration consistency verification method provided in the above-described method embodiments.

[0170] This application also provides a computer program product, which includes a computer program executed by a processor to implement the data migration consistency verification method provided in the above-described method embodiments.

[0171] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0172] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data migration consistency verification method, characterized in that, The method includes: According to the preset partitioning strategy, the source data set of the source database and the target data set of the target database are divided into multiple data blocks, and the data range identification information corresponding to each data block is recorded. For each data block, the field order of each record in each data block is adjusted according to a preset field order rule, and the format of each field value is standardized according to a preset field type standardization rule. Based on the adjusted and standardized data content, the hash value of each data block is determined. The source hash set, composed of the hash values ​​of data blocks in the source data set, and the target hash set, composed of the hash values ​​of data blocks in the target data set, are sorted respectively. Then, the data blocks with inconsistent hash values ​​are marked as difference blocks by comparing them bit by bit according to the same sequence position after sorting. For any discrepancy block, the data records indicated by the data range identifier information of the discrepancy block are compared one by one to determine the data records with discrepancies.

2. The method according to claim 1, characterized in that, The process of dividing the source data set of the source database and the target data set of the target database into multiple data blocks according to a preset block division strategy includes: The source data set and the target data set are divided into multiple data blocks according to a preset row count threshold; or, The source data set and the target data set are divided into multiple data blocks according to a preset data volume threshold; or, The amount of data in each data record in the source data set and the target data set is detected. If the amount of data in a single data record in the current data block exceeds a preset threshold, the number of records contained in the current data block is reduced.

3. The method according to claim 1 or 2, characterized in that, The step of adjusting the field order of each record within each data block according to a preset field order rule includes: Based on the metadata of the source data tables in the source database, determine the global field sorting sequence; Sort each field of each record in each data block according to the global field sorting sequence.

4. The method according to claim 3, characterized in that, The step of determining the global field sorting sequence based on the metadata of the source data tables in the source database includes: Extract the field names and field numbers from the source data table as a baseline template; The fields in the target data table of the target database are matched with the benchmark template in a case-insensitive manner according to the field names; Fields that match successfully are mapped to their corresponding positions in the global field sorting sequence according to the field numbers indicated in the baseline template, and fields that fail to match are marked as unmatched fields.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Filter out internal fields other than hidden fields and auto-incrementing primary keys, so that the hidden fields and internal fields do not participate in field order adjustment and hash value calculation.

6. The method according to any one of claims 1 to 5, characterized in that, The process of standardizing the format of each field value according to the preset field type standardization rules includes any one of the following: If the field value is a numeric type, the field value will be uniformly converted into a decimal string representation, leading zeros will be removed, and a preset number of decimal places will be retained; If the field value is of type time, then the field value will be uniformly converted into a date and time string in a preset format; If the field value is a string, remove leading and trailing whitespace characters from the field value and replace empty strings or NULL values ​​with preset placeholders; If the field value is of type JSON, then the keys in the JSON object of the field value are sorted alphabetically and extra whitespace characters are removed; If the field value is a binary large field type, then the binary data of the field value will be uniformly converted into a Base64 encoded string.

7. The method according to any one of claims 1 to 6, characterized in that, The process of determining the hash value of each data block based on the adjusted and unified data content includes: Following the adjusted field order, the unified field values ​​are concatenated sequentially, and a preset unique separator is inserted between adjacent field values ​​to obtain the original text block of each data block. The original text block of each data block is hashed to obtain the hash value of each data block.

8. The method according to any one of claims 1 to 7, characterized in that, For any discrepancy block, the process of comparing each data record according to the data range identifier information of the discrepancy block to determine the data records with discrepancies includes: Obtain the number of the difference blocks; If the number is less than a preset threshold, then each data record indicated by the data range identifier information corresponding to the difference block is read concurrently, and the content is compared one by one to determine the data records with differences. If the number is greater than or equal to the preset threshold, the continuous block interval where the difference block is located is recursively split into two parts. After determining the difference sub-intervals, the data records in the difference sub-intervals are compared one by one to determine the data records with differences.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Output a structured difference report, which includes at least one of the following: difference block identifier, primary key range of difference data records, difference row number, difference field name, field value in the source database, and field value in the target database.

10. A data migration consistency verification device, characterized in that, The device includes: The partitioning module is used to divide the source data set of the source database and the target data set of the target database into multiple data blocks according to a preset partitioning strategy, and record the data range identification information corresponding to each data block. The first determining module is used to adjust the field order of each record in each data block according to a preset field order rule, and to perform format unification processing on each field value according to a preset field type standardization rule, and to determine the hash value of each data block based on the adjusted and unified data content; The comparison module is used to sort the source hash set composed of the hash values ​​of data blocks in the source data set and the target hash set composed of the hash values ​​of data blocks in the target data set, and compare them bit by bit according to the same sequence position after sorting, and mark the data blocks with inconsistent hash values ​​as difference blocks. The second determining module is used to compare the data records one by one according to the data range identification information of the difference block for any difference block, and determine the data records with differences.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the data migration consistency verification method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program, which is loaded and executed by a processor to implement the data migration consistency verification method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program executed by a processor to implement the data migration consistency verification method as described in any one of claims 1 to 9.

Images