Hash bucket-based data comparison method, device, equipment and medium

Abstract

The application relates to the technical field of data comparison, and discloses a data comparison method and device based on hash bucketing, equipment and a medium, which comprises the following steps: in response to a comparison instruction, obtaining source-side data files to be compared and target-side data files to be compared; based on the source-side data files to be compared and the target-side data files to be compared, determining a bucketing key field; calculating a hash value according to the bucketing key field, performing a modulo operation on the hash value according to a preset number of buckets, obtaining a bucket number of each record, writing each record into a corresponding bucket file according to the corresponding bucket number, and obtaining source-side data in a bucket and target-side data in a bucket; performing a differential comparison on the source-side data in the bucket and the target-side data in the bucket, and obtaining difference records; classifying and counting the difference records according to difference types, and generating a data comparison report. Through hash bucketing and pointer comparison, the application helps users to intuitively display difference details, and improves the efficiency of data migration verification and repair.

Description

Technical Field

[0001] This invention relates to the field of data comparison technology, and in particular to a data comparison method, apparatus, device and medium based on hash bucketing. Background Technology

[0002] In today's big data era, data migration is becoming increasingly frequent. Ensuring the accuracy and consistency of data during the migration process has become a critical issue that urgently needs to be addressed. To address this, data migration comparison tools have emerged. These tools can perform detailed comparisons between source and target data and automatically generate discrepancy results, effectively reducing the risk of misjudgment caused by manual judgment. These tools significantly improve the efficiency of comparison work, reduce migration risks, and provide reliable assurance for the smooth progress of data migration.

[0003] However, current mainstream data comparison methods still primarily rely on direct inter-database comparisons, which involve directly executing queries between the source and target databases to perform the comparison. While this method offers good real-time performance, its drawbacks are also significant. Query operations place additional performance pressure on both the source and target databases, especially when dealing with extremely large datasets, easily impacting the normal operation of business systems. Furthermore, the comparison process is time-consuming, consumes substantial system resources, and is generally inefficient, failing to meet the timeliness requirements of large-scale data migration. In addition, due to inherent differences in character sorting rules, data write order, storage engines, and optimizer behavior among different databases, direct comparisons often lead to inconsistent results, further weakening the accuracy and reliability of the comparisons. Therefore, there is an urgent need to explore more efficient and robust comparison schemes to address increasingly complex data migration scenarios. Summary of the Invention

[0004] The main objective of this invention is to provide a data comparison method, apparatus, device, and storage medium based on hash bucketing, aiming to solve the technical problems of existing data comparison methods having a significant impact on database performance in data migration scenarios, low comparison efficiency, and inaccurate results due to differences in heterogeneous databases.

[0005] To achieve the above objectives, the present invention provides a data comparison method based on hash bucketing, comprising: In response to the comparison command, the source database and the target database respectively retrieve the source database and the target database to be compared data file; Based on the source and target data files to be compared, determine the key fields for bucketing; A hash algorithm is used to calculate the hash value in the source and target data files to be compared based on the key bucket fields. The hash value is then moduloed according to the preset number of buckets to obtain the bucket number of each record. Records with the same bucket number are written into the corresponding source and target bucket files to obtain the data in the source bucket and the data in the target bucket. Set source pointers and target pointers for the data in the source bucket and the data in the target bucket respectively. Perform a difference comparison on the data in the source bucket and the data in the target bucket based on the source pointers and the target pointers to obtain difference records. The difference records are classified and statistically analyzed according to the difference type, and a data comparison report is generated.

[0006] Furthermore, to achieve the above objectives, the present invention provides a data comparison device based on hash bucketing, comprising: The file export module is used to retrieve the source data file to be compared and the target data file to be compared from the source database and the target database, respectively, in response to the comparison command. The bucketing field module is used to determine the key fields for bucketing based on the source data file to be compared and the target data file to be compared. The data bucketing module is used to calculate the hash value in the source and target data files to be compared based on the key bucketing fields using a hash algorithm, and to perform a modulo operation on the hash value according to the preset number of buckets to obtain the bucket number of each record. Records belonging to the same bucket number are written into the corresponding source bucket file and target bucket file respectively to obtain the data in the source bucket and the data in the target bucket. The difference comparison module is used to set source pointers and target pointers for the data in the source bucket and the data in the target bucket, respectively, and to perform a difference comparison on the data in the source bucket and the data in the target bucket based on the source pointers and the target pointers to obtain difference records. The comparison report module is used to classify and statistically analyze the difference records according to the difference type and generate a data comparison report.

[0007] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a hash-based data comparison program stored in the memory and executable on the processor, wherein when the hash-based data comparison program is executed by the processor, it implements the steps of the hash-based data comparison method as described above.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a hash-based data comparison program, wherein the hash-based data comparison program, when executed by a processor, implements the steps of the hash-based data comparison method as described above.

[0009] Beneficial Effects: This invention relates to the field of data comparison technology and discloses a data comparison method, apparatus, device, and medium based on hash bucketing, comprising: responding to a comparison command, obtaining source-end comparison data files and target-end comparison data files from a source-end database and a target-end database, respectively; determining bucketing key fields based on the source-end comparison data files and target-end comparison data files; using a hash algorithm to calculate hash values ​​in the source-end comparison data files and target-end comparison data files according to the bucketing key fields, and performing a modulo operation on the hash values ​​according to a preset number of buckets to obtain a bucket number for each record; writing records belonging to the same bucket number into the corresponding source-end bucket file and target-end bucket file, respectively, to obtain data within the source-end bucket and data within the target-end bucket; setting source-end pointers and target-end pointers for the data within the source-end bucket and the data within the target-end bucket, respectively; performing a difference comparison on the data within the source-end bucket and the data within the target-end bucket according to the source-end pointers and the target-end pointers to obtain difference records; classifying and statistically analyzing the difference records according to the difference type, and generating a data comparison report. This invention utilizes efficient hash bucketing and parallel processing to decompose source and target data into buckets and compare them one by one. It accurately identifies three types of discrepancies: missing data in the source, missing data in the target, and inconsistent content. It automatically generates a categorized statistical report to help users intuitively grasp the data consistency status, quickly locate problem records, and improve the efficiency of data migration verification and maintenance. Attached Figure Description

[0010] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a data comparison method based on hash bucketing in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the data comparison method based on hash bucketing of the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the hash-based data comparison device of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0011] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0012] The data comparison method based on hash bucketing provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server, triggered by the client, retrieves the source and target comparison data files from the source and target databases, respectively. Based on these files, it determines key bucketing fields. A hash algorithm is used to calculate the hash values ​​in both files based on these key fields, and a modulo operation is performed on the hash values ​​according to a preset number of buckets to obtain a bucket number for each record. Records belonging to the same bucket number are written to their corresponding source and target bucket files, resulting in source and target bucket data. Source and target pointers are set for the source and target bucket data, respectively. A difference comparison is performed on the source and target bucket data based on these pointers to obtain difference records. Finally, the difference records are categorized and statistically analyzed according to their difference types to generate a data comparison report. This invention utilizes efficient hash bucketing and parallel processing to decompose source and target data into buckets and compare them bucket by bucket. It accurately identifies three types of discrepancies: missing data in the source, missing data in the target, and inconsistent content. It automatically generates categorized statistical reports, helping users intuitively understand data consistency status, quickly locate problem records, and improve the efficiency of data migration verification and maintenance. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0013] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the hash-based data comparison method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0014] like Figure 2 As shown, the data comparison method based on hash bucketing proposed in this invention includes the following steps: S100: In response to the comparison command, retrieve the source-side comparison data file and the target-side comparison data file from the source-side database and the target-side database, respectively; S200. Based on the source and target data files to be compared, determine the key fields for bucketing. S300. Using a hash algorithm, calculate the hash values ​​in the source-end data file to be compared and the target-end data file to be compared according to the bucket key fields, and perform a modulo operation on the hash values ​​according to the preset number of buckets to obtain the bucket number of each record. Write the records belonging to the same bucket number into the corresponding source-end bucket file and target-end bucket file respectively to obtain the data in the source-end bucket and the data in the target-end bucket. S400: Set source pointers and target pointers for the data in the source bucket and the data in the target bucket, respectively; perform a difference comparison on the data in the source bucket and the data in the target bucket based on the source pointers and the target pointers to obtain difference records; S500. Classify and statistically analyze the difference records according to the difference type, and generate a data comparison report.

[0015] In this embodiment, the data to be compared is exported from both the source and target ends. This data is saved in a structured file format, such as a comma-separated values ​​file (CSV). The file should contain key fields for subsequent bucketing and unique identification of records, typically the primary key or composite primary key of the database table. The export process first verifies the connection permissions of both databases to ensure data query and export capabilities. Then, based on the comparison requirements, the export range can be selected as the entire table, a specified field, or filtered by conditions. The export is executed, and file verification is completed to confirm that the file can be opened normally, the fields are complete, and the number of data rows meets expectations, thereby avoiding comparison errors caused by export mistakes.

[0016] After obtaining the data files to be compared from the source and target ends, the system needs to determine the key fields used for bucketing. The selection of this field directly affects the balance and accuracy of hash bucketing: if the database table has a primary key, the primary key field is selected first; if a unique index field exists, that unique index is selected; if neither exists, a single field or a combination of fields with a distinguishability of more than 70% is selected as the bucketing field. This selection strategy ensures that each record can be mapped to the corresponding bucket as evenly and uniquely as possible during bucketing.

[0017] After determining the key fields for bucketing, the system uses a hash algorithm to calculate the hash value for each record in the source and target data files. MurmurHash, a non-cryptographic hash function, is preferred due to its high performance, low collision rate, and uniform distribution, making it ideal for bucketing large-scale data. Then, based on the preset number of buckets N, a modulo operation is performed on the hash value calculated for each record. That is, the bucket number equals the hash value modulo N, thus assigning each record a bucket number from zero to N minus one. Subsequently, each record in the source data file is written to its corresponding source bucket file according to its bucket number, for example, named "source_bucket_0", "source_bucket_1", up to "source_bucket_N-1"; the target data file undergoes the same bucketing operation, generating corresponding target bucket files, such as "target_bucket_0", "target_bucket_1", etc. Each bucket file stores the complete data fields of all records belonging to that bucket, and the source and target sides use the same bucketing logic to ensure that records with the same primary key or unique identifier will fall into the bucket with the same number, laying the foundation for subsequent detailed comparison.

[0018] The core of the data comparison phase lies in performing a row-by-row comparison between the source and target data within each bucket. The system iterates through each bucket number from zero to N minus one, first checking if both the source and target bucket files exist simultaneously. If one is missing, the bucket is marked as a difference bucket, and it is recorded whether it exists only in the source or only in the target. For buckets where both exist, the system loads the source and target bucket files into memory respectively, and then uses a dual-axis quicksort algorithm to sort the data in both memory locations in ascending order to eliminate interference caused by inconsistent storage order at the underlying database level. After sorting, two pointers are set, pointing to the starting positions of the source and target buffers after sorting, respectively, and then streaming comparison begins: the data records currently pointed to by the two pointers are compared sequentially. If the data are completely equal, both pointers move forward one position simultaneously; if the data are not equal, the difference is recorded. Specifically, when one pointer points to null or the difference field happens to be the bucketing key field, it is determined to be a missing record that exists only on the source end or only on the target end, and only the pointer on the corresponding end is moved down; when the difference field is not a bucketing key field, it is determined to be a record with inconsistent content on both ends, and both pointers are moved down simultaneously. This process is repeated until both pointers point to null, at which point the comparison of that bucket ends. By summarizing the comparison results of all buckets, three types of differences can be obtained: missing records on the source end (i.e., the target end has them but the source end does not), missing records on the target end (i.e., the source end has them but the target end does not), and records with inconsistent data content (i.e., the key field exists but the content of other fields is different).

[0019] In the process of differential identification, this invention specifically addresses the problem of misjudgment caused by traditional comparisons between heterogeneous databases due to differences in data types, storage semantics, and physical formats, and establishes a unified normalization adaptation scheme. Regarding null value handling, Oracle databases strictly distinguish between empty strings and null values ​​(NULL), automatically converting empty strings to null values, while MySQL allows both to coexist. Therefore, traditional direct comparisons produce a large number of false differences. This invention performs a unified conversion logic on all character and text fields during the data preprocessing stage, semantically aligning the null values ​​converted by default in Oracle and PostgreSQL with the null strings in MySQL. It also adds a null value flag, discards the original field values, and uniformly classifies them into three states: "valid data," "empty string," and "true null value." Only when the null value states or valid content at both ends are inconsistent is it determined to be a true data difference. For numeric types, different databases have different storage formatting rules for high-precision numeric, floating-point, and decimal fields. For example, MySQL's Decimal type automatically retains valid trailing zeros, while Oracle automatically truncates invalid trailing zeros. Furthermore, the precision truncation thresholds and rounding rules for floating-point types also differ. This invention designs a unified and standardized logic for numeric types. First, it standardizes the number of digits retained at both ends of the numeric value based on the precision configuration of the field metadata. It then performs uniform rounding and truncation on excessively long decimal places. Next, it automatically removes meaningless invalid zeros at the end of the numeric value. Finally, it only verifies the core valid value of the numeric value, ignoring formatting differences such as trailing zeros or precision padding differences. For time types, Oracle's date format has fixed precision while timestamp supports microseconds, and PostgreSQL supports multiple time zone time types. This leads to inconsistencies in millisecond padding, format separators, time zone offsets, and precision digits across different databases for the same business time. This invention uniformly converts all time type fields to millisecond-level timestamps under the Coordinated Universal Time (UTC) standard for comparison, while masking differences in millisecond zero padding, time separator format, and local time zone offset. It performs consistency checks only based on absolute time values, thereby completely eliminating the differences in time storage format and time zone between heterogeneous databases.

[0020] Finally, the system categorizes and statistically analyzes all identified discrepancies according to their types, automatically summarizing and generating a structured data comparison report. This report not only clearly shows whether the overall data is consistent, but also accurately locates the specific bucket to which the discrepancy record belongs and the record itself, distinguishing between different types such as missing records, newly added records, and records with inconsistent content. Users can further trigger repair operations based on the discrepancy information in the report. The system can generate Structured Query Language (SQL) statements to be executed and provide visual feedback on the repair results, thus achieving a complete automated closed loop from problem discovery to problem location and then to problem repair.

[0021] In one embodiment, S100 includes: S101. In response to the comparison instruction, the comparison instruction is parsed to obtain comparison information and connection information; S102. Establish a communication connection between the source database and the target database based on the connection information; S103. Perform authorization verification on the source database and the target database based on the comparison information; S104. When the source database and the target database pass the authorization verification, obtain the source database to be compared and the target database to be compared.

[0022] In this embodiment, upon receiving a comparison command triggered by the user, the system first parses the command to extract two types of key information: one is comparison information, including the name of the data table to be compared, the list of fields to be exported, filtering conditions (such as a specific time range or business type), and the desired export file format; the other is connection information, including the network address, port number, database instance name, authentication credentials (username and password), and connection timeout parameters of the source and target databases. Based on the parsed connection information, the system uses the corresponding database driver (e.g., Java Database Connectivity (JDBC)) to establish stable network communication links with the source and target databases respectively, ensuring that subsequent data reading operations can be executed normally. After the connection is established, the system performs authorization verification on both databases according to the operation scope specified in the comparison information, that is, checks whether the current connected user has the necessary permissions for data querying and exporting, such as whether they have the SELECT permission to select a specified table, and whether they are allowed to read system tables or perform export operations; only after both the source and target have passed authorization verification and confirmed that the permissions are correct will the system officially begin to obtain the data to be compared. The data acquisition process does not load all records into memory at once. Instead, it follows the export strategy configured in the comparison information (such as exporting the entire table, exporting specified fields, or exporting by conditional filtering). By executing Structured Query Language (SQL) queries, data that meets the criteria is read from both databases in a streaming or batch manner and saved as preset structured file formats (such as comma-separated values ​​(CSV) files). This prepares the raw data files for subsequent hash bucketing and bucket-by-bucket comparison. This pre-process ensures operational security, avoiding task anomalies due to insufficient permissions or connection failures, and also adapts to the comparison needs of different business scenarios through flexible export configurations.

[0023] In one embodiment, S104 includes: S1041. When the source database and the target database pass the authorization verification, the comparison instruction is parsed to obtain the comparison range; S1042. Based on the comparison range, extract the source-end comparison data and the target-end comparison data from the source-end database and the target-end database, respectively. S1043. Export the source-end comparison data and the target-end comparison data in a preset structured file format as a source-end comparison data file and a target-end comparison data file; S1044. Perform preliminary verification on the source-end data file to be compared and the target-end data file to be compared. S1045. If there are no abnormalities in the source data file to be compared and the target data file to be compared, the preliminary verification is successful.

[0024] In this embodiment, after both the source and target databases have passed authorization verification, the system further parses the comparison command issued by the user to extract the specific comparison scope for this comparison task. This comparison scope includes the database table names to be compared, the list of fields to be exported (selecting all fields from the entire table or only a portion of specified fields), and optional filtering conditions. For example, the WHERE clause in Structured Query Language (SQL) can be used to limit the export to a subset of data within a specific time range, for a specific business type, or that meets other business rules. This design makes the comparison task highly flexible; users do not need to export the entire massive table but can tailor the data size as needed, thereby reducing the amount of data processed subsequently.

[0025] Based on the comparison range obtained from the parsing, the system performs corresponding data query operations in both the source and target databases to extract the data records to be compared. The extraction process does not load all data into memory at once, but rather uses streaming or batch fetching to reduce memory pressure on the database runtime and avoid holding lock resources for extended periods, which could affect normal access to the business system. The extracted source and target data are then exported as separate files according to a preset structured file format: the source comparison data file and the target comparison data file. The system supports multiple export formats, such as comma-separated values ​​(CSV) or Excel, etc. During export, it is essential to ensure that the files contain the key fields used for bucketing and uniquely identifying records, and that both ends use the same field order and encoding rules to provide a consistent data foundation for subsequent hash bucketing and row-by-row comparison.

[0026] After the export operation is completed, the system does not immediately enter the bucketing stage. Instead, it first performs a preliminary verification on the two generated data files. This verification mainly checks whether the files can be opened normally, whether the number of fields and columns is complete, whether there are garbled characters or character encoding problems, and whether the number of data rows in the files matches the expected number of exported data. By quickly verifying the file integrity, readability, and basic row count, it can effectively identify data incompleteness or format errors caused by abnormal situations such as network interruption, permission changes, insufficient disk space, or database query timeouts during the export process. Only when neither the source nor the target data file to be compared has the above-mentioned abnormalities is the preliminary verification deemed successful. The system then confirms that the data preparation stage has been successfully completed and can proceed to the subsequent hash bucketing and comparison processing stages. This pre-verification mechanism greatly avoids the spread of hidden problems introduced during the export stage to the subsequent complex comparison process, thereby ensuring the reliability of the entire comparison task and the validity of the results.

[0027] In one embodiment, S200 includes: S201. Perform data preprocessing on the source-end data file to be compared and the target-end data file to be compared, respectively, to obtain standardized source-end data file to be compared and standardized target-end data file to be compared. S202. Obtain the standardized table structure information of the source and target data files to be compared, and select logical bucket keys based on the table structure information. S203. Verify the actual distinguishability of the logical bucket key in the source data file to be compared and the target data file to be compared, respectively. S204. When the actual distinguishability of the logical bucketing key in both the source end and the target end data files to be compared reaches a preset threshold, the logical bucketing key is determined as a bucketing key field. S205. If the actual distinguishability of the logical bucket key in either the source end data file to be compared or the target end data file to be compared does not reach the preset threshold, then other logical bucket keys are selected again for distinguishability verification until a bucket key field that meets the conditions is selected.

[0028] In this embodiment, before formally performing hash bucketing, data preprocessing operations must be performed on the exported source and target data files to eliminate incomparability caused by differences in the storage format of the source database. The preprocessing covers multiple dimensions: for date or time data, its format is uniformly converted to ensure that both ends use the same year-month-day or timestamp representation; for numeric data, it is processed with uniform precision, and the same rounding and truncation rules are applied to excessively long decimal places; irrelevant leading or trailing spaces or invalid characters in character fields are removed, and null or default values ​​are normalized and categorized according to predetermined rules. After the above cleaning and standardization transformation, the system obtains standardized source and target data files, making the data at both ends basically comparable at the field level.

[0029] After standardization, the system reads the table structure information corresponding to both files and obtains the candidate field information (i.e., source-side unique identifiers) used to uniquely identify records from the source data file. Then, combining this information with the table structure of the target file, a preliminary logical bucketing key is selected. The selection strategy follows a clear priority: if the database table has a primary key field, that primary key is selected as the logical bucketing key; if a primary key does not exist but a unique index field exists, that unique index is used; if neither exists, a substitute key needs to be found from a single field or a combination of multiple fields, such as selecting a combination of fields with high business distinguishability. This initial selection of the logical bucketing key is based solely on the table structure definition and its effectiveness in actual data needs further verification.

[0030] To ensure that subsequent hash bucketing distributes data evenly across buckets and avoids a large number of records being concentrated in a few buckets, which could lead to an unbalanced comparison load, the system must rigorously verify the actual distinguishability of the initially selected logical bucketing key in both the source and target data files. Distinguishing ability is typically measured by calculating the percentage of unique values ​​for that field or combination of fields in their respective datasets (i.e., the ratio of cardinality to the total number of rows). Only when the actual distinguishability in both the source and target datasets reaches a preset distinguishability threshold (e.g., a distinguishability rate of 70% or higher) can the logical bucketing key be officially determined as the final bucketing key field for subsequent hash value calculation and modulo bucketing operations.

[0031] Conversely, if the discrimination of either the source or target dataset fails to reach the preset threshold, it indicates that the candidate field contains a large number of duplicate values ​​in the dataset at one end, making it impossible to effectively disperse the data. In this case, the system will automatically discard the current candidate key and select another field or combination of fields as the new logical bucketing key, and perform the same bidirectional discrimination check on the new candidate key again. This process iterates until a qualified bucketing key field with sufficient discrimination in both the source and target datasets is successfully selected. Through this rigorous verification and reselection mechanism, the system fundamentally ensures the scientific nature and adaptability of bucketing key selection, laying a solid foundation for efficient parallel comparison of large-scale data in the future.

[0032] In one embodiment, S300 includes: S301. Based on the total amount of data to be compared and the available computing resources, set the number of buckets; S302. Read the source data file and the target data file to be compared line by line, and extract the key fields for bucketing for each record; S303. Calculate the hash value of the key field of the bucket using a hash algorithm, and perform a modulo operation on the number of buckets based on the hash value to obtain the bucket number; S304. Write all records belonging to the same bucket number in the source-end data file to be compared into the same source-end bucket file to obtain the data in the source-end bucket. S305. Write all records belonging to the same bucket number in the target end data file to be compared into the same target end bucket file to obtain the data in the target end bucket.

[0033] In this embodiment, after determining the final key fields for bucketing, a reasonable number of buckets needs to be set based on the total size of the data to be compared and the currently available computing resources (such as the number of CPU cores, available memory size, number of parallel processing threads, etc.). Selecting the number of buckets N is a trade-off: too small a value for N will result in excessively large data volumes in each bucket, leading to excessively high peak consumption when loading a single bucket into memory, making it difficult to leverage the advantages of parallel processing; too large a value for N will generate too many bucket files, increasing file read / write overhead and task scheduling costs. Typically, the system estimates the expected number of records a single bucket can hold based on the total number of record rows and sets N to an appropriate value based on the number of CPU cores, for example, making the number of buckets an integer multiple of the number of CPU cores, to fully utilize parallel computing capabilities while ensuring that the data volume of each bucket can be processed efficiently in memory.

[0034] Once the number of buckets is determined, the system begins the formal data bucketing operation. This process requires reading the source and target data files to be compared line by line, extracting the corresponding bucketing key fields (i.e., the previously verified primary key, unique index, or combination of high-discrimination fields) for each record in the files. Subsequently, the system uses an efficient hash algorithm—preferably MurmurHash (a non-cryptographic hash function known for its high performance, low collision rate, and excellent uniform distribution characteristics)—to calculate an integer hash value for the bucketing key fields of each record. This hash value is a value that is uniformly distributed over a large range, and it is strongly correlated with the content of the record; the same field value will produce completely identical hash values ​​at both the source and target ends.

[0035] After obtaining the hash value, the system performs a modulo operation on it with the preset number of buckets N. This means the bucket number equals the remainder of the hash value divided by N, thus assigning each record a bucket number within the range of zero to N minus one. Due to the deterministic nature of the modulo operation, any record with the same bucket key field value (regardless of whether it comes from the source or target) will be assigned the exact same bucket number. This is the prerequisite for subsequent precise intra-bucket comparison of the data from both ends. Based on the calculated bucket number, the system writes each record from the source data file to be compared into the corresponding source bucket file, for example, named "source_bucket_0.avro", "source_bucket_1.avro", up to "source_bucket_N-1.avro". Each bucket file only stores the complete data fields of all records belonging to that bucket number. Simultaneously, the same operation is performed on the target data file to be compared: reading line by line, extracting the same bucket key field, calculating the hash value and performing a modulo operation to obtain the bucket number, and then writing the record into the corresponding target bucket file, such as "target_bucket_0.avro", "target_bucket_1.avro", etc. In this way, all records on both the source and target sides are distributed into N independent bucket file pairs according to a unified hash bucketing strategy. Records contained in the corresponding bucket files of any given bucket number on both the source and target sides will have their bucketing key fields belonging to the same hash value range. This ensures that logically corresponding data fragments can be directly compared during subsequent bucket-by-bucket comparisons. The entire writing process can employ batch appending or streaming writing methods to improve input / output (I / O) efficiency. Simultaneously, each bucket file is stored independently, without interference, creating conditions for subsequent parallel loading and comparison.

[0036] In one embodiment, S400 includes: S401. For each bucket number, sort the data in the source bucket and the data in the target bucket according to the bucket key field to obtain the source sorted data and the target sorted data. S402. Set the source pointer and target pointer of the source sorting data and the target sorting data respectively; S403. Traverse the source-end sorted data and the target-end sorted data, and perform a difference comparison between the current record pointed to by the source pointer and the current record pointed to by the target pointer: S404. If the current record pointed to by the source pointer and the current record pointed to by the target pointer are the same, then move the source pointer and the target pointer one position to the right. S405. If the source pointer is null and the target pointer is not null, it is determined that there is a difference record in the record pointed to by the target pointer, and the target pointer is moved one position to the right. S406. If the target pointer is empty while the source pointer is not empty, it is determined that there is a difference record in the record pointed to by the source pointer, and the source pointer is moved one position to the right. S407. Repeat the difference comparison process until both the source pointer and the target pointer point to the end of their respective sorted data to obtain the difference record.

[0037] In this embodiment, for the data in the source bucket and the target bucket corresponding to each bucket number, the system first sorts the data at both ends according to the determined bucketing key fields. Since the data at both ends may come from different database systems, their underlying storage order naturally differs (e.g., insertion order, physical storage order, or different index organization methods). Direct row-by-row comparison would produce a large number of false differences due to disordered order. Therefore, the system uses a dual-axis quicksort algorithm or other efficient sorting methods to sort the data in the source bucket and the target bucket loaded into memory in ascending order, ensuring that the data at both ends has a consistent logical order under the same sorting rules. After sorting, the source-end sorted data and the target-end sorted data are obtained. At this point, the data at both ends are arranged in ascending order according to the value of the same bucketing key field, providing an ordered foundation for subsequent streaming comparison.

[0038] Subsequently, the system sets up a pointer for the source sorted data and a pointer for the target sorted data, namely the source pointer and the target pointer. Both pointers initially point to the first record (i.e., the starting position) in their respective sorted data. The system then iterates through the source and target sorted data, retrieving the record currently pointed to by the source pointer and the target pointer respectively in each round of comparison, and performing a field-by-field difference comparison. The comparison includes not only the key fields for bucketing but also all non-key fields (such as ordinary attribute columns) in the record that need to be compared, to ensure the integrity and consistency of the data content.

[0039] The comparison process follows clear rules: If the record currently pointed to by the source pointer and the record currently pointed to by the target pointer are completely identical in all comparison fields, then the record is determined to be corresponding and identical at both ends. In this case, the system moves both the source pointer and the target pointer one position forward and continues to compare the next record. If the source pointer has reached the end of its sorted data (i.e., is empty), while the target pointer has not reached the end (is not empty), it means that there is extra data on the target end that does not have a corresponding record on the source end. The system marks this record as a record with a difference on the target end (i.e., a missing record on the source end) and moves the target pointer one position forward to continue checking the next record on the target end. Conversely, if the target pointer is empty and the source pointer is not empty, it means that there is extra data on the source end that does not have a corresponding record on the target end. The system marks this record as a record with a difference on the source end (i.e., a missing record on the target end) and moves the source pointer one position forward.

[0040] The above comparison and pointer movement process is repeated continuously until both the source and target pointers point to the end of their sorted data (i.e., both pointers are null), indicating that all records in the bucket at both ends have been traversed. Through this streaming comparison strategy based on sequential scanning with two pointers after sorting, the system can efficiently identify differences in all records within a single bucket in linear time complexity, avoiding the quadratic performance overhead of nested loop comparisons. Simultaneously, it accurately distinguishes between three categories: consistent records, records existing only at the source end, and records existing only at the target end, providing a complete and accurate set of difference records for subsequent difference classification statistics and report generation.

[0041] In one embodiment, S500 includes: S501. According to the difference type, summarize the source missing records, target missing records and inconsistent records in detail to obtain the difference details. S502. Display the difference types and difference details on the user interface; S503. Generate a structured query statement based on the difference types and details in the data comparison report; S504. Obtain the repair instruction triggered by the user in the user interface, execute the structured query statement in response to the repair instruction, and feed back the execution result to the user interface.

[0042] In this embodiment, after completing the comparison of all buckets and collecting the difference records, the system first classifies and summarizes all difference records according to the difference type, generating a structured difference detail. The difference types are divided into three categories: the first category is "missing records at the source," which are records that exist at the target but not at the source, usually indicating that extra data was added to the target during data migration; the second category is "missing records at the target," which are records that exist at the source but not at the target, indicating that some data failed to be successfully written to the target during migration; the third category is "inconsistent records," which are records that exist at both ends (corresponding through key bucket fields), but differ in other non-key fields such as values, text, or time attributes, indicating that the data was changed or corrupted during migration. The system establishes a detailed list for each type of difference. Each detailed record includes the complete field values, the bucket number, the specific difference field names, and a comparison of the difference content, allowing users to accurately grasp the details of each difference point.

[0043] The system then visualizes the categorized and summarized difference types and their corresponding details through the user interface. The display can use tables, category labels, or tree structures, allowing operations personnel to quickly see the overall data consistency status and delve into the specific record details under each difference category. The interface also provides filtering, sorting, and search functions, enabling users to quickly locate difference records of interest, such as filtering by table name, bucket number, or difference field.

[0044] Based on the identified discrepancy types and details in the data comparison report, the system can automatically generate Structured Query Language (SQL) statements to repair these discrepancies. Different types of discrepancies are addressed with different repair statements: for "missing records on the target side" (i.e., records present on the source side but absent on the target side), an INSERT statement is generated to add the missing data from the source side to the target side; for "missing records on the source side" (i.e., records present on the target side but absent on the source side), a DELETE statement is generated to remove the redundant data from the target side; for "inconsistent records," an UPDATE statement is generated to update the inconsistent fields in the target side using the values ​​of the corresponding fields from the source side. The generated SQL statements fully consider the syntax characteristics of the target database and are organized according to transaction rules to ensure the atomicity and rollbackability of the repair operations.

[0045] Finally, the system provides users with an entry point for performing repairs on the user interface, such as displaying a preview of the SQL statement to be executed, along with "Execute Repair" or "Confirm Repair" buttons. When a user triggers a repair command on the interface, the system responds by executing the repair operation line by line or in batches according to the generated SQL statements. During execution, the system captures the execution status (success or failure) of each SQL statement and records the reasons for failure (such as violation of unique constraints, data type mismatch, or insufficient permissions). After execution, the system provides real-time feedback on the repair results—including the number of successfully repaired records, the number of failed records, and detailed reasons for failure—to the user interface in the form of a visual report. Users can clearly see which tables' discrepancies have been successfully repaired, which repairs failed, and the specific reasons for the failures through the monitoring interface, thus enabling timely manual intervention or adjustment of the repair strategy. This closed-loop process achieves a complete automated chain from discrepancy discovery, discrepancy display, automatic generation of repair scripts to repair execution and result feedback, greatly improving the efficiency of data migration verification and error correction.

[0046] In one embodiment, a hash-based data comparison device is provided, which corresponds one-to-one with the hash-based data comparison method described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the hash-based data comparison device of the present invention. The modules include a root directory determination module 10, a traversal filtering module 20, an attribute reading module 30, a data conversion module 40, a sorting module 50, and a report generation module 60. Detailed descriptions of each functional module are as follows: The file export module 10 is used to retrieve the source data file to be compared and the target data file to be compared from the source database and the target database, respectively, in response to the comparison command. Bucketing Field Module 20 is used to determine the key fields for bucketing based on the source data file to be compared and the target data file to be compared. The data bucketing module 30 is used to calculate the hash value in the source end data file to be compared and the target end data file to be compared according to the bucketing key field using a hash algorithm, and to perform a modulo operation on the hash value according to the preset number of buckets to obtain the bucket number of each record. Records belonging to the same bucket number are written into the corresponding source end bucket file and target end bucket file respectively to obtain the data in the source end bucket and the data in the target end bucket. The difference comparison module 40 is used to set source pointers and target pointers for the data in the source bucket and the data in the target bucket respectively, and to perform a difference comparison on the data in the source bucket and the data in the target bucket according to the source pointers and the target pointers to obtain difference records; The comparison report module 50 is used to classify and statistically analyze the difference records according to the difference type and generate a data comparison report.

[0047] In one embodiment, the file export module 10 includes: In response to a comparison command, the comparison command is parsed to obtain comparison information and connection information; A communication connection is established between the source database and the target database based on the connection information; Authorization verification is performed on the source database and the target database based on the comparison information. When the source database and the target database pass the authorization verification, the source database and the target database are obtained for comparison.

[0048] In one embodiment, the file export module 10 includes: When the source database and the target database pass the authorization verification, the comparison instruction is parsed to obtain the comparison range; Based on the comparison range, source-side comparison data and target-side comparison data are extracted from the source-side database and the target-side database, respectively. The source-end comparison data and the target-end comparison data are exported in a preset structured file format as source-end comparison data file and target-end comparison data file; A preliminary verification is performed on the source and target data files to be compared. If there are no anomalies in the source and target data files to be compared, the preliminary verification is successful.

[0049] In one embodiment, the bucketing field module 20 includes: Data preprocessing is performed on the source data file to be compared and the target data file to be compared, respectively, to obtain standardized source data file to be compared and standardized target data file to be compared; Obtain the standardized table structure information of the source and target data files to be compared, and select logical bucketing keys based on the table structure information; The actual distinguishability of the logical bucket key in the source and target data files to be compared is verified respectively. When the actual distinguishability of the logical bucketing key in both the source and target data files to be compared reaches a preset threshold, the logical bucketing key is determined as the bucketing key field. If the actual distinguishability of the logical bucket key in either the source data file to be compared or the target data file to be compared fails to reach the preset threshold, then other logical bucket keys are selected for distinguishability verification until a bucket key field that meets the conditions is selected.

[0050] In one embodiment, the data bucketing module 30 includes: The number of buckets is set based on the total amount of data to be compared and the available computing resources; Read the source and target data files to be compared line by line, and extract the key fields for bucketing for each record; The hash value of the key field of the bucket is calculated by a hash algorithm, and the bucket number is obtained by performing a modulo operation on the number of buckets based on the hash value. Write all records belonging to the same bucket number in the source-end data file to be compared into the same source-end bucket file to obtain the data in the source-end bucket. Write all records belonging to the same bucket number in the target end's data file to be compared into the same target end bucket file to obtain the data within the target end bucket.

[0051] In one embodiment, the difference comparison module 40 includes: For each bucket number, sort the data in the source bucket and the data in the target bucket according to the bucketing key fields to obtain the source sorted data and the target sorted data. Set the source pointer and target pointer for the source sorted data and the target sorted data respectively; Iterate through the source-side and target-side sorted data, and perform a difference comparison between the current record pointed to by the source pointer and the current record pointed to by the target pointer: If the current record pointed to by the source pointer and the current record pointed to by the target pointer are the same, then the source pointer and the target pointer are moved one position to the right. If the source pointer is null while the target pointer is not null, it is determined that there is a difference record in the record pointed to by the target pointer, and the target pointer is moved one position to the right. If the target pointer is null while the source pointer is not null, it is determined that there is a difference record in the record pointed to by the source pointer, and the source pointer is moved one position to the right. Repeat the difference comparison process until both the source pointer and the target pointer point to the end of their respective sorted data, thus obtaining the difference record.

[0052] In one embodiment, the comparison report module 50 includes: Based on the type of difference, detailed summaries were made for missing records at the source end, missing records at the target end, and records with inconsistent content, resulting in a detailed difference list. The difference types and difference details are displayed in the user interface; Based on the difference types and details in the data comparison report, generate a structured query statement; Obtain the repair command triggered by the user in the user interface, execute the structured query statement in response to the repair command, and feed back the execution result to the user interface.

[0053] For specific limitations regarding the hash-based data comparison device, please refer to the aforementioned limitations on the hash-based data comparison method, which will not be repeated here. Each module in the aforementioned hash-based data comparison device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0054] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a hash-based data comparison method on the server side.

[0055] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a hash-based data comparison method.

[0056] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: In response to the comparison command, the source database and the target database respectively retrieve the source database and the target database to be compared data file; Based on the source and target data files to be compared, determine the key fields for bucketing; A hash algorithm is used to calculate the hash value in the source and target data files to be compared based on the key bucket fields. The hash value is then moduloed according to the preset number of buckets to obtain the bucket number of each record. Records with the same bucket number are written into the corresponding source and target bucket files to obtain the data in the source bucket and the data in the target bucket. Set source pointers and target pointers for the data in the source bucket and the data in the target bucket respectively. Perform a difference comparison on the data in the source bucket and the data in the target bucket based on the source pointers and the target pointers to obtain difference records. The difference records are classified and statistically analyzed according to the difference type, and a data comparison report is generated.

[0057] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: In response to the comparison command, the source database and the target database respectively retrieve the source database and the target database to be compared data file; Based on the source and target data files to be compared, determine the key fields for bucketing; A hash algorithm is used to calculate the hash value in the source and target data files to be compared based on the key bucket fields. The hash value is then moduloed according to the preset number of buckets to obtain the bucket number of each record. Records with the same bucket number are written into the corresponding source and target bucket files to obtain the data in the source bucket and the data in the target bucket. Set source pointers and target pointers for the data in the source bucket and the data in the target bucket respectively. Perform a difference comparison on the data in the source bucket and the data in the target bucket based on the source pointers and the target pointers to obtain difference records. The difference records are classified and statistically analyzed according to the difference type, and a data comparison report is generated.

[0058] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0059] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0061] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0062] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data comparison method based on hash bucketing, characterized in that, Includes the following steps: In response to the comparison command, the source database and the target database respectively retrieve the source database and the target database to be compared data file; Based on the source and target data files to be compared, determine the key fields for bucketing; A hash algorithm is used to calculate the hash value in the source and target data files to be compared based on the key bucket fields. The hash value is then moduloed according to the preset number of buckets to obtain the bucket number of each record. Records with the same bucket number are written into the corresponding source and target bucket files to obtain the data in the source bucket and the data in the target bucket. Set source pointers and target pointers for the data in the source bucket and the data in the target bucket respectively. Perform a difference comparison on the data in the source bucket and the data in the target bucket based on the source pointers and the target pointers to obtain difference records. The difference records are classified and statistically analyzed according to the difference type, and a data comparison report is generated.

2. The data comparison method based on hash bucketing as described in claim 1, characterized in that, In response to the comparison command, the source database and the target database respectively retrieve the source database and the target database, including: In response to a comparison command, the comparison command is parsed to obtain comparison information and connection information; A communication connection is established between the source database and the target database based on the connection information; Authorization verification is performed on the source database and the target database based on the comparison information. When the source database and the target database pass the authorization verification, the source database and the target database are obtained for comparison.

3. The data comparison method based on hash bucketing as described in claim 2, characterized in that, When the source database and the target database pass the authorization verification, they obtain the source database data to be compared and the target database data to be compared, including: When the source database and the target database pass the authorization verification, the comparison instruction is parsed to obtain the comparison range; Based on the comparison range, source-side comparison data and target-side comparison data are extracted from the source-side database and the target-side database, respectively. The source-end comparison data and the target-end comparison data are exported in a preset structured file format as source-end comparison data file and target-end comparison data file; A preliminary verification is performed on the source and target data files to be compared. If there are no anomalies in the source and target data files to be compared, the preliminary verification is successful.

4. The data comparison method based on hash bucketing as described in claim 1, characterized in that, Based on the source and target data files to be compared, the key fields for bucketing are determined, including: Data preprocessing is performed on the source data file to be compared and the target data file to be compared, respectively, to obtain standardized source data file to be compared and standardized target data file to be compared; Obtain the standardized table structure information of the source and target data files to be compared, and select logical bucketing keys based on the table structure information; The actual distinguishability of the logical bucket key in the source and target data files to be compared is verified respectively. When the actual distinguishability of the logical bucketing key in both the source and target data files to be compared reaches a preset threshold, the logical bucketing key is determined as the bucketing key field. If the actual distinguishability of the logical bucket key in either the source data file to be compared or the target data file to be compared fails to reach the preset threshold, then other logical bucket keys are selected for distinguishability verification until a bucket key field that meets the conditions is selected.

5. The data comparison method based on hash bucketing as described in claim 1, characterized in that, A hash algorithm is used to calculate the hash values ​​in the source and target data files to be compared based on the key bucket fields. The hash values ​​are then modulo-operated according to a preset number of buckets to obtain the bucket number for each record. Records belonging to the same bucket number are written into the corresponding source and target bucket files, respectively, to obtain the data within the source and target buckets, including: The number of buckets is set based on the total amount of data to be compared and the available computing resources; Read the source and target data files to be compared line by line, and extract the key fields for bucketing for each record; The hash value of the key field of the bucket is calculated by a hash algorithm, and the bucket number is obtained by performing a modulo operation on the number of buckets based on the hash value. Write all records belonging to the same bucket number in the source-end data file to be compared into the same source-end bucket file to obtain the data in the source-end bucket. Write all records belonging to the same bucket number in the target end's data file to be compared into the same target end bucket file to obtain the data within the target end bucket.

6. The data comparison method based on hash bucketing as described in claim 1, characterized in that, Set source pointers and target pointers for the data in the source bucket and the data in the target bucket, respectively. Based on these pointers, perform a difference comparison between the data in the source bucket and the data in the target bucket to obtain difference records, including: For each bucket number, sort the data in the source bucket and the data in the target bucket according to the bucketing key fields to obtain the source sorted data and the target sorted data. Set the source pointer and target pointer for the source sorted data and the target sorted data respectively; Iterate through the source-side and target-side sorted data, and perform a difference comparison between the current record pointed to by the source pointer and the current record pointed to by the target pointer: If the current record pointed to by the source pointer and the current record pointed to by the target pointer are the same, then the source pointer and the target pointer are moved one position to the right. If the source pointer is null while the target pointer is not null, it is determined that there is a difference record in the record pointed to by the target pointer, and the target pointer is moved one position to the right. If the target pointer is null while the source pointer is not null, it is determined that there is a difference record in the record pointed to by the source pointer, and the source pointer is moved one position to the right. Repeat the difference comparison process until both the source pointer and the target pointer point to the end of their respective sorted data, thus obtaining the difference record.

7. The data comparison method based on hash bucketing as described in claim 1, characterized in that, The difference records are categorized and statistically analyzed according to their difference types, and a data comparison report is generated, including: Based on the type of difference, detailed summaries were made for missing records at the source end, missing records at the target end, and records with inconsistent content, resulting in a detailed difference list. The difference types and difference details are displayed in the user interface; Based on the difference types and details in the data comparison report, generate a structured query statement; Obtain the repair command triggered by the user in the user interface, execute the structured query statement in response to the repair command, and feed back the execution result to the user interface.

8. A data comparison device based on hash bucketing, characterized in that, The hash-based data comparison device includes: The file export module is used to retrieve the source data file to be compared and the target data file to be compared from the source database and the target database, respectively, in response to the comparison command. The bucketing field module is used to determine the key fields for bucketing based on the source data file to be compared and the target data file to be compared. The data bucketing module is used to calculate the hash value in the source and target data files to be compared based on the key bucketing fields using a hash algorithm, and to perform a modulo operation on the hash value according to the preset number of buckets to obtain the bucket number of each record. Records belonging to the same bucket number are written into the corresponding source bucket file and target bucket file respectively to obtain the data in the source bucket and the data in the target bucket. The difference comparison module is used to set source pointers and target pointers for the data in the source bucket and the data in the target bucket, respectively, and to perform a difference comparison on the data in the source bucket and the data in the target bucket based on the source pointers and the target pointers to obtain difference records. The comparison report module is used to classify and statistically analyze the difference records according to the difference type and generate a data comparison report.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a hash-based data comparison program stored in the memory and executable on the processor, wherein the hash-based data comparison program, when executed by the processor, implements the steps of the hash-based data comparison method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a hash-based data comparison program, which, when executed by a processor, implements the steps of the hash-based data comparison method as described in any one of claims 1-7.

Images