Data verification method and device in big data cluster migration, equipment and medium

Abstract

The invention relates to the technical field of big data, and provides a data verification method and device in big data cluster migration, equipment and a medium, and the method comprises the steps: obtaining a source data cluster and a target data cluster; dividing the source data cluster and the target data cluster into a plurality of data layers according to a standard data warehouse layering strategy, and binding different data comparison strategies for each data layer; establishing a fingerprint baseline table and constructing a sampling model feature library; performing data fragmentation processing on each data layer, and controlling each data layer to execute a data comparison task in parallel; and if any data layer of the target data cluster and any data layer of the source data cluster are detected to have difference, recording difference data and performing data restoration. According to the method, the source data cluster and the target data cluster are subjected to hierarchical parallel fragmentation comparison, a plurality of fragments of each layer are subjected to data comparison tasks in parallel at the same time, different comparison strategies are bound to the data layers, the comparison process can be achieved, and the overall data processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of big data cluster technology, and in particular to a data verification method, apparatus, device and medium for big data cluster migration. Background Technology

[0002] With the advent of the big data era, the amount of data that enterprises need to process and analyze is growing exponentially. To meet the needs of business expansion, cost control, or technology upgrades, big data cluster migration (such as migrating from on-premises data centers to cloud platforms, or from one cloud service provider to another) has become commonplace. However, data migration is not simply a data transfer process. Its success hinges not only on the complete transfer of data from the source cluster to the target cluster, but also on ensuring that the migrated data maintains consistency, availability, and reliability—that is, ensuring data integrity and consistency. Data verification is a crucial step in ensuring these standards are met.

[0003] Currently, there are relevant solutions for data verification after big data cluster migration, such as full data comparison: after migration, all data in the source data cluster and the target data cluster are compared one by one to ensure that the data is completely consistent; incremental data comparison: only the incremental data generated during the migration process is compared and verified; sampling verification: by randomly selecting a part of the data for comparison, to verify the consistency and integrity of the data; and tool-based verification: using automated tools for data migration and verification.

[0004] While the methods described above have met the data verification requirements to some extent, they still have certain shortcomings, such as insufficient comprehensiveness: existing verification methods may only cover some data types or scenarios, failing to fully cover all possible data problems, such as complex data structures or dynamically changing data; some verification methods are inefficient, especially when processing large-scale data, which may lead to excessively long migration times and affect business continuity; existing verification tools may require manual operation or script writing, lacking intelligence and automation, and when migrating between different platforms or clusters, existing verification methods may not be able to effectively handle differences in data format and structure. Summary of the Invention

[0005] This invention provides a data verification method, apparatus, device, and medium for big data cluster migration, in order to solve the problems of low efficiency, incomplete coverage, and high resource consumption faced in big data cluster migration scenarios with large scale, multi-source heterogeneity, and high real-time requirements.

[0006] Firstly, a data verification method for big data cluster migration is provided, including:

[0007] Obtain the source data cluster and the target data cluster;

[0008] According to the standard data warehouse layering strategy, the source data cluster and the target data cluster are divided into multiple data layers, and different data comparison strategies are bound to each data layer.

[0009] Perform data sharding on each of the data layers and control each data layer to execute the data comparison task in parallel.

[0010] If a difference is detected between any data layer of the target data cluster and the source data cluster, the difference data is recorded and data repair is performed.

[0011] Secondly, a data verification device for big data cluster migration is provided, including:

[0012] The data acquisition module is used to acquire source data clusters and target data clusters;

[0013] The layering module is used to divide the source data cluster and the target data cluster into multiple data layers according to the standard data warehouse layering strategy, and bind different data comparison strategies to each data layer.

[0014] The comparison module is used to perform data fragmentation processing on each of the data layers and control each data layer to execute the data comparison task in parallel.

[0015] The repair module is used to record the difference data and perform data repair when a difference is detected in any data layer of the target data cluster and the source data cluster.

[0016] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data verification method in the above-mentioned big data cluster migration.

[0017] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the data verification method described above in the big data cluster migration.

[0018] In the data verification method, apparatus, computer equipment, and storage medium of the present invention for big data cluster migration, when performing data comparison, the source data cluster and the target data cluster are compared in a hierarchical and parallel manner, allowing multiple fragments in each layer to perform data comparison tasks simultaneously and in parallel, thereby improving the data comparison speed and reducing the data comparison time. In addition, by binding different comparison strategies to each data layer, the data loading, transformation, and storage processes can be optimized, thereby improving the overall data processing efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an application environment for the data verification method in big data cluster migration according to an embodiment of the present invention;

[0021] Figure 2 This is a flowchart illustrating the data verification method in big data cluster migration according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the data cluster layering and sharding processing flow according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the business data source layer data comparison process according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the data comparison process in the data detail layer according to an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the data comparison process in the data aggregation layer according to an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of the data comparison process at the data indicator layer in an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of priority queue sorting according to an embodiment of the present invention;

[0028] Figure 9 This is a schematic diagram of the automatic repair strategy according to an embodiment of the present invention;

[0029] Figure 10 This is a schematic diagram of the data verification device structure in the big data cluster migration according to an embodiment of the present invention;

[0030] Figure 11 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0031] Figure 12 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] The data verification method for big data cluster migration provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain source and target data clusters; divide the source and target data clusters into multiple data layers according to a standard data warehouse layering strategy, and bind different data comparison strategies to each data layer; perform data sharding processing on each data layer, and control each data layer to execute data comparison tasks in parallel; if a difference is detected in any data layer of the target and source data clusters, the difference data is recorded and data repair is performed. In this invention, when performing data comparison, by performing layered and parallel sharding comparison of the source and target data clusters, multiple shards in each layer can perform data comparison tasks simultaneously and in parallel, thereby improving the data comparison speed and reducing the data comparison time; in addition, by binding different comparison strategies to each data layer, the data loading, transformation, and storage processes can be optimized, improving the overall data processing efficiency. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The present invention will now be described in detail through specific embodiments.

[0034] Figure 2 This is a flowchart illustrating a data verification method for big data cluster migration provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the data verification method in big data cluster migration according to an embodiment of the present invention includes the following steps:

[0035] S1, obtain the source data cluster and the target data cluster.

[0036] A data cluster refers to a physical or logical entity formed by integrating dispersed data resources (such as databases, file systems, and data lakes) through technical means. During data migration, the source data cluster and the target data cluster are the two endpoints of the migration process, representing the starting point of data migration and the ending point of data migration, respectively. The source data cluster is the cluster where the data is currently located before migration; it is the physical or logical storage location of the original data and may include legacy systems, older platform versions, or data centers that need to be decommissioned. The target data cluster is the new storage environment for the data after migration, typically an upgraded system. The data involved in the source and target data clusters can be transaction data, payment data, business data, or purchase data in the fintech field, or medical data in the medical technology field, such as personal health records, prescriptions, and examination reports. This embodiment does not specifically limit the data involved in the source and target data clusters.

[0037] S2, the source data cluster and the target data cluster are divided into multiple data layers according to the standard data warehouse layering strategy, and different data comparison strategies are bound to each data layer.

[0038] A standard data warehouse layering strategy divides the data warehouse into multiple layers according to the data processing and usage stages. Each layer has its specific responsibilities and data characteristics. In this embodiment of the invention, the source data cluster and the target data cluster are divided into four data layers according to the standard data warehouse layering strategy: Data Detail Layer (ODS), Data Detail Layer (DWD), Data Summary Layer (DWS), and Data Index Layer (ADS). By dividing the source data cluster and the target data cluster into multiple data layers, the data organization in the data warehouse becomes clearer, and the data at different levels has clear responsibilities and uses, making it easier for data developers and managers to classify, manage, maintain, and update the data.

[0039] For each of the aforementioned data layers, different data comparison strategies can be bound to it. Each layer's data can be processed using different methods and comparison strategies based on its characteristics, optimizing the data loading, transformation, and storage processes and improving overall data processing efficiency. A comparison strategy refers to the specific methods, rules, and logic used when comparing data to determine whether the data in two data sources (source data cluster and target data cluster) are consistent, and how to perform difference analysis and processing when they are inconsistent. Different comparison strategies are suitable for different types of data and business scenarios.

[0040] S3, perform data sharding processing on each data layer, and control each data layer to execute the data comparison task in parallel.

[0041] As mentioned earlier, the source data cluster and the target data cluster are divided into multiple data layers. Furthermore, each layer of the source data cluster and the target data cluster can be sharded, that is, the data layer is split into multiple data shards, allowing multiple shards in each layer to perform data comparison tasks in parallel, thereby improving the overall comparison efficiency.

[0042] S4. If a difference is detected between any data layer of the target data cluster and the source data cluster, record the difference data and perform data repair.

[0043] When comparing data at each layer of the source data cluster and the target data cluster, if any difference is found in the data at any layer, the difference data is recorded, such as the content, location, time, type, etc., and the data in the target data cluster is corrected to be consistent with the state of the source data cluster.

[0044] This invention provides a data verification method for big data cluster migration. When performing data comparison, the source data cluster and the target data cluster are compared in a hierarchical and parallel manner, allowing multiple shards in each layer to perform data comparison tasks simultaneously and in parallel, thereby improving the data comparison speed and reducing the data comparison time. In addition, by binding different comparison strategies to each data layer, the data loading, transformation and storage process can be optimized, thereby improving the overall data processing efficiency.

[0045] In an optional embodiment of the present invention, after obtaining the source data cluster and the target data cluster in step S1 above, metadata preparation can also be performed. That is, before data comparison, the digital fingerprints corresponding to each data layer are calculated, and fingerprint baselines are generated; feature information of data tables in the source data cluster is collected, and the feature information is processed by feature engineering to generate a sampling model feature library for use by the intelligent sampling model; the feature information includes data ontology features, business logic features, historical difference features, and / or system metadata features.

[0046] In other words, a fingerprint baseline table can be established first. This table records the fingerprint baseline version for each data layer. Specifically, the corresponding XXHASH3_128 values ​​are calculated for the data before the source and target data clusters are compared, and then stored in the fingerprint baseline table. In this embodiment, a comparison baseline table is established to record the fingerprint baseline versions of each table in the source and target data clusters. This table is used to calculate fingerprints for the entire dataset during the initial runtime, recording the baseline fingerprints of the source and target data clusters. By comparing the current fingerprint with the baseline fingerprint, it is quickly determined whether the data has changed. Subsequent incremental updates are used to maintain the fingerprints (e.g., using a rolling hash algorithm) to confirm the incremental fingerprints.

[0047] In addition, a sampling model feature library can be constructed. In this embodiment, the feature collector can extract statistical information, content features, business rule engine, data dictionary and other data ontology features, business logic features, historical difference features and system metadata features from the source data table. After feature engineering processing, these features are used to drive the intelligent sampling model.

[0048] The feature collector can be a small program or SQL script used to acquire data features from the source data cluster, including data ontology features, business logic features, historical difference features, and / or system metadata features. It also establishes metrics for each metadata element, such as field change frequency, data business weight, and historical difference distribution. Field change frequency can be collected from table change logs to statistically analyze the change frequency of each field. Data business weight can be manually confirmed based on data business attributes. Historical difference distribution can be statistically analyzed by comparing differences between historical table field data.

[0049] Data ontology features are used to describe the original information of a row of data or a field, and can include statistical information (such as the number of rows), content features (length, distribution, LOB fingerprint, etc.) and other content information.

[0050] Business logic features refer to the rules and semantics that business requirements dictate and that data must meet. These can include features such as business rule engines and data dictionaries.

[0051] Historical difference characteristics, such as the number of times the field / table differs during comparison, the difference rate, the difference type, and other difference information.

[0052] System metadata features are information such as the location, version, and resource consumption of recorded data.

[0053] The extracted features can be used to build a sampling model feature library, which can then be used to train an intelligent sampling model. The intelligent sampling model in this embodiment is a machine learning model that learns features such as data distribution, business rules, and historical differences. Using this intelligent sampling model, key fields can be adaptively extracted from sharded data during the data comparison process, improving the quality of data comparison.

[0054] The intelligent sampling model in this embodiment can be a dual model that combines an isolated forest with an LSTM.

[0055] Input to the intelligent sampling model: Compare the following features in the table: field change frequency (number of modifications in the last 7 days), data business weight (regulatory field weight × 1.5), and historical difference distribution (field-level difference heatmap).

[0056] Intelligent sampling model output: List of high-risk sample IDs (confidence > 85%).

[0057] For the output results of the dual models, samples are divided into high-risk and low-risk samples, and different verification strategies are used for samples of different levels; high-risk samples: full field row-by-row comparison; low-risk samples: 5% of the samples are sampled for parallel field verification.

[0058] In an optional embodiment of the present invention, metadata corresponding to the source data cluster and the target data cluster can be collected separately; based on the metadata corresponding to the source data cluster and the target data cluster, the table structures of the source data cluster and the target data cluster are checked to see if they are consistent. If they are consistent, the process continues; if they are inconsistent, the comparison task is paused and a repair operation is performed.

[0059] This embodiment first performs a rapid structural comparison of the metadata of the source and target data clusters to ensure complete consistency in fields, types, primary keys, and partitions before proceeding to row-level data comparison. This avoids misjudgments or omissions caused by field misalignment, implicit conversions, or missing columns. When collecting metadata from the source and target data clusters, Hive metadata can be exported to a JSON file (and Python scripts can be used to compare metadata differences), outputting and marking missing databases, tables, or partitions in the target data cluster.

[0060] The following section provides a detailed explanation of the specific comparison process.

[0061] Overall, for both the source and target data clusters, hierarchical sharding can be performed. Specifically, this can be done through a hierarchical sharding controller, followed by a corresponding comparison strategy based on the sharding results. Finally, after aggregation analysis of the comparisons, discrepancy repair is performed. Figure 3 As shown.

[0062] I. Layered processing

[0063] Step S2 above mentions dividing the source data cluster and the target data cluster into multiple data layers according to a standard data warehouse layering strategy. In an optional embodiment of the present invention, the source data cluster and the target data cluster can be divided into a business data source layer, a data detail layer, a data aggregation layer, and a data indicator layer according to a standard data warehouse layering strategy, and different comparison strategies are bound to different layers. The data layers and the bound comparison strategies are as follows:

[0064] 1. The business data source layer (ODS layer) is bound to an incremental fingerprint comparison strategy, which can capture data changes in real time.

[0065] 2. Data Detail Layer (DWD layer), which is bound to an intelligent sampling strategy, can dynamically identify high-risk fields.

[0066] 3. The Data Summarization Layer (DWS layer) binds key indicator fingerprints and sampling coordination strategies, which can determine key indicator fingerprints and high-risk sample sampling.

[0067] 4. Data Metrics Layer (ADS Layer): Binds golden indicator fingerprints and business-sensitive sample sampling strategies.

[0068] II. Segmentation Processing

[0069] Furthermore, data sharding is performed on each data layer, as follows:

[0070] 1. The business data source layer (ODS layer) divides tables into shards based on data volume and field attributes. Tables containing LOB (Large Object, referring to extremely large, unstructured, or semi-structured data stored in the database) fields are sharded separately and field-level fingerprints are enabled. This ensures even sharding while accommodating differences in comparison complexity due to field characteristics, achieving optimal parallel efficiency. For example, large tables (>100 million rows) are split using ranges (by primary key, timestamp, or business incrementing key) (each shard contains 1 million to 5 million rows) to avoid excessively large single tasks. Medium-sized tables (10 million to 100 million rows) are directly sharded on a single table or a combination of time partitioning and range sharding, resulting in moderate task granularity. Small tables (<10 million rows) are not sharded; the entire table is used for the task, reducing scheduling overhead.

[0071] For field attribute dimensions, these can include high cardinality unique keys, low cardinality enumeration fields, large fields / LOBs, and frequently updated time fields. By using these field attribute dimensions for data layer sharding, changes are concentrated within a single shard, reducing cross-shard comparisons. This achieves load balancing and maximizes resource utilization during the parallel comparison phase, while ensuring the accuracy and low latency of difference detection.

[0072] In this embodiment, for tables containing LOB (Large Object) fields, independent sharding combined with field-level fingerprinting can reduce the time spent on LOB table comparison.

[0073] 2. The Data Detail Layer (DWD layer) is split according to the partition dimensions of the comparison table. This means a large table can be vertically divided into several sub-tables for parallel comparison, thereby reducing the data volume per task and shortening the overall processing time. Specific split dimensions can include time, region, and business line dimensions. The time dimension can be daily, weekly, monthly, or hourly, depending on the source table's partition key. The region dimension can include provinces, cities, etc. The business line dimension can be such as product lines, channels, partners, etc.

[0074] In this embodiment, hotspot partitions (changes > 100,000 times / day) can be automatically split into sub-partitions.

[0075] 3. Data aggregation layer (DWS layer), which is allocated according to subject domains and splits high-frequency indicators into independent shards; the subject domains include customers, transactions and / or products.

[0076] The customer domain can include members, accounts, customer profiles, and customer behavior logs; the transaction domain can include orders, payments, refunds, settlements, and statements; and the product domain can include product master data, SKUs, prices, inventory, and assets.

[0077] Optionally, high-frequency metrics such as GMV (Gross Merchandise Value) and user retention rate can be split into independent segments.

[0078] 4. Data indicator layer (ADS layer): It is divided and categorized according to report units, with the gold indicator being divided into separate segments.

[0079] For example, the ADS layer can be divided into reporting units such as the balance sheet, risk exposure statement, income statement, and cash flow statement.

[0080] Gold indicators can be segmented separately. Gold preparation refers to the actual fields in the table, which can be key regulatory indicators such as regulatory reporting, capital measurement, and risk weighting, to avoid omissions during data comparison.

[0081] III. Fragmentation Verification

[0082] The above step S3 controls each data layer to perform the data comparison task in parallel, including: verifying each fragment using the comparison strategy bound to each data layer. The comparison process of each data layer is described below.

[0083] 1. Business Data Source Layer (ODS Layer): For each data shard, an incremental fingerprint is calculated and compared. If the fingerprint difference exceeds a preset threshold, intelligent sampling is performed for supplementary inspection. If intelligent sampling identifies a high-risk sample, a full-field hash comparison is performed on that high-risk sample. That is, if... Figure 4 As shown, the data comparison process of the business data source layer includes the following steps S3-11 to S3-13.

[0084] S3-11, calculate incremental fingerprints for each data shard and perform fingerprint comparison. A sliding window can capture the changed data of the source data cluster and the target data cluster within 24 hours. For the obtained changed data, calculate digital fingerprints (such as XXHASH3_128 fingerprints), synthesize a global rolling fingerprint, and realize the comparison of incremental fingerprints between the source data cluster and the target data cluster. That is, by performing shard hash calculation on the changed data within the sliding window to obtain incremental fingerprints, all incremental fingerprints are aggregated in an orderly manner to generate a global fingerprint (such as sorted by shard ID): global fingerprint = Hash(shard 1 fingerprint || shard 2 fingerprint || ... || shard N fingerprint). Based on the above scheme, the recalculation of the entire data can be avoided (time complexity O(1)).

[0085] S3-12, If the fingerprint difference exceeds a preset threshold, intelligent sampling supplementary inspection is performed. The preset threshold in this embodiment can be set according to different requirements; here, it is preferably 0.001%. If the fingerprint difference is ≤0.001%, it is considered an acceptable error, and no further action is required. If the fingerprint difference exceeds 0.001%, the intelligent sampling model is triggered to perform supplementary inspection.

[0086] S3-13 If intelligent sampling discovers a high-risk sample, then perform a full-field hash comparison on the high-risk sample.

[0087] The intelligent sampling model can extract samples from discrepancy shards according to the risk level of each field, marking samples containing fields with a confidence level ≥ 85% as high-risk samples. Only samples containing high-risk fields undergo row-by-row, full-field hash verification to accurately locate discrepancy rows. Data in the source / target data cluster is then repaired manually or via script based on the discrepancy results. Due to inconsistencies in fingerprint comparison, the fingerprint baseline table corresponding to the target data cluster must be updated after manual discrepancy repair.

[0088] This embodiment can also be configured to resume fingerprint transmission from the most recent segment after an abnormal interruption in fingerprint generation.

[0089] 2. Data Detail Layer (DWD Layer): Utilizes an intelligent sampling model to perform intelligent sampling based on each data shard. If an anomaly is found, a fingerprint check is performed on that anomaly. If an anomaly is found in the fingerprint, a deep validation of business rules is triggered. That is, for example... Figure 5 As shown, the data comparison process of the business data source layer includes the following steps S21 to S23.

[0090] S21, use an intelligent sampling model to perform intelligent sampling based on the data in each shard; run the intelligent sampling model on each shard to extract key fields. The intelligent sampling model in this embodiment is designed to efficiently extract representative samples from massive amounts of data. During sampling, random sampling, stratified sampling (based on data shards, time partitions, etc.), or more advanced model-based sampling (e.g., adjusting the sampling ratio based on data distribution characteristics) can be used. Specifically, Spark's `sample` function or Hive's `TABLESAMPLE` can be used, or custom sampling logic can be defined.

[0091] S22, if an abnormal field is found, perform fingerprint re-check on the abnormal field. If a specific abnormal field is marked in the sampling results. If an abnormal field exists in the sampling results, only the above-mentioned abnormal field will have its fingerprint regenerated for supplementary checking. The abnormal field detection in this embodiment identifies potentially abnormal fields in the sampled data. It can be identified through preset rules (such as comparison of strings between the source data cluster and the target data cluster, abnormal proportion of null values, abnormal value range, abnormal format, etc.) or machine learning models (such as anomaly detection models). For example, the value of a certain field in the source data cluster is 'a' and the value in the target data cluster is 'b', which are not equal; or the value of a certain field suddenly becomes all 0 (which does not conform to business norms).

[0092] Fingerprint re-checking involves performing a more comprehensive examination of a field (not just the sample, but potentially the entire dataset across a specific shard) after it has been flagged as an anomaly. Specifically, this involves calculating the fingerprint of that field. A fingerprint typically refers to a hash value, such as MD5, SHA-1, or a faster non-cryptographic hash (like xxHash) used to hash all values ​​of the entire field. Note that when calculating a field's fingerprint, all values ​​must be hashed in a specific order (e.g., sorted by primary key) to ensure repeatability. Optionally, a full fingerprint calculation for the field is only triggered when an anomaly is detected during sampling, thus avoiding the overhead of a full calculation.

[0093] S23. If an anomaly is detected in the fingerprint, a deep validation of the business rules is triggered. Deep validation of business rules refers to performing a line-by-line, full-field check of the discrepancy fields according to the business rules to confirm the actual discrepancies. For the discrepancy data, discrepancy details can be recorded and output for subsequent repair and tracking.

[0094] In other words, when fingerprint re-examination reveals an anomaly (i.e., the calculated fingerprint does not match the expected fingerprint), it indicates that the data in that field does indeed have a problem and requires further investigation. At this point, deep validation of business rules can be initiated, executing predefined business rule validations. These rules may be complex SQL queries, UDF functions, or calls to external rule engines. For example, validating whether the value of a certain field conforms to business logic (such as order amount cannot be negative, user age must be within a reasonable range, etc.).

[0095] In other words, if intelligent sampling detects an anomaly, the abnormal field is verified again by fingerprint comparison. If the fingerprint comparison is also abnormal, manual business verification is performed.

[0096] 3. Data Summarization Layer (DWS Layer): Calculates the digital fingerprint for each data shard. If fingerprint differences exist, intelligent sampling is performed on the corresponding data shard. If high-risk dimensions are found in the sampling results, correlation verification is conducted. That is, for example... Figure 6As shown, the data comparison process of the business data source layer includes the following steps S31 to S33.

[0097] S31 calculates the digital fingerprint corresponding to each data segment. It can calculate the XXHASH64 fingerprint of the indicator value (such as the SUM (sales) fingerprint).

[0098] S32, if fingerprint differences exist, intelligent sampling will be performed on the corresponding data segments. That is, when the digital fingerprints are inconsistent, the intelligent sampling model will be triggered for sampling verification.

[0099] S33. If high-risk dimensions are found in the sampling results, then perform association verification. Association verification can be performed on the upstream and downstream associations of high-risk dimensions (such as primary key-foreign key, summary-detail, indicator-dimension) to check for any broken associations or inconsistent values.

[0100] For example, based on the business characteristics of the comparison table, different dimensions such as high-risk and low-risk can be divided. High-risk dimensions (such as customer risk level) are fully verified, while low-risk dimensions (such as static regional information) are sampled at 1%.

[0101] When the sampling model is abnormal or there are discrepancies in the sampling data comparison, it will automatically upgrade to a full field comparison. You can manually write SQL to repair the differences between the current cluster tables, or synchronize data from the source data cluster to the target data cluster.

[0102] 4. Data Metric Layer (ADS Layer): Calculates the golden indicator fingerprint corresponding to the report data and performs golden fingerprint comparison. If a difference is found, sensitive field sampling is performed. If a logical error is found, a backtracking of the data aggregation layer is triggered. That is, for example... Figure 7 As shown, the data comparison process of the business data source layer includes the following steps S41 to S43.

[0103] S41, calculate the golden indicator fingerprint corresponding to the report data and perform golden fingerprint comparison. When calculating the golden fingerprint, you can calculate the fingerprint of the XXHASH64 table corresponding to the ADS layer indicator (e.g., debt-to-asset ratio = (Assets - Liabilities) / Assets). The fingerprint synthesis period can be a fixed time, such as 1:00 AM every day.

[0104] S42, if a difference is found, sensitive field sampling is performed. Sensitive fields can be pre-defined high-risk fields (such as negative values ​​or fields with abnormal fluctuations) that undergo 100% validation, while low-risk fields (such as static configuration fields) are sampled at 0.1%.

[0105] S43. If a logical error is detected, a backtracking process at the data aggregation layer is triggered. Errors at the logical level located during sampling, such as incorrect formula calculations, inconsistent definitions, or discrepancies between summaries and details, automatically trigger a backtracking process from the ADS layer to the upper-level DWS layer for data comparison. For intelligent sampling data comparison, if discrepancies are found in the compared fields, business experts are notified for secondary confirmation.

[0106] As mentioned earlier, each data layer can execute comparison tasks in parallel. In an optional embodiment of the present invention, different comparison strategies can be set for each data layer, as shown in Table 1.

[0107] Table 1

[0108] Data layer Verification type priority Business data source layer (ODS layer) Fingerprint comparison > Fragment verification Data Detail Layer (DWD Layer) Smart Sampling > Field Validation Data aggregation layer (DWS layer) Fingerprint + Sampling Collaboration Data Metrics Layer (ADS Layer) Gold Index Fingerprint > Report Sampling

[0109] This embodiment sets scheduling priorities for each data layer, so that when cluster resources are scarce, high-priority tasks get CPU / memory first, saving resources and costs.

[0110] In an optional embodiment of the present invention, a priority queue is set up. If comparison resources are insufficient, comparison tasks are executed according to the priority queue order. The priority queue includes high priority, medium priority, and low priority. By setting different priority queues (high, medium, and low), when comparison resources are insufficient, the high-priority queue runs comparison tasks first. Figure 8 As shown, high priority can include ADS report verification (golden indicator), medium priority can include DWD intelligent sampling, and low priority can include ODS fingerprint verification. Once an anomaly is detected at the ADS layer, DWD deep verification is immediately triggered, realizing a linkage mechanism of "automatically falling back one layer after a problem is detected" to prevent errors from continuing to spread downstream.

[0111] In an optional embodiment of the present invention, a resource degradation strategy can also be set to automatically and dynamically adjust the comparison strategy according to the resource load of the target data cluster. For example, if the resource load of the target data cluster is 80%, a sampling strategy is enabled; if the resource load of the target data cluster is 90%, the fingerprint comparison strategy is downgraded to an incremental fingerprint strategy; if the resource load of the target data cluster is 95%, only the golden indicator fingerprint is verified.

[0112] Furthermore, this embodiment can also automatically switch strategies, such as enabling the fingerprint comparison strategy by default. If the resource utilization rate is <70%, a collaborative strategy combining fingerprint and sampling is enabled; if the resource utilization rate is <70%, a sampling-only strategy is enabled.

[0113] This embodiment prioritizes the most critical ADS reports, enabling rapid error detection, especially under resource constraints. Setting different priorities ensures accuracy while reducing resource utilization. Furthermore, resource degradation strategies allow for the rational allocation of resource usage.

[0114] IV. Difference Handling and Optimization

[0115] The difference classification and repair scheme of this invention is as follows:

[0116] If data is missing, the missing fragment is located and retransmitted using incremental fingerprinting; if fields are inconsistent, the intelligent sampling model is used to determine whether they are high-risk fields; if there are abnormal indicators, fingerprint differences trigger a full backtracking at the DWS layer; if the report is incorrect, abnormal golden indicators trigger manual review at the ADS layer.

[0117] Automatic repair strategies can be such as Figure 9 As shown, the differences between the source and target data clusters are categorized. If the difference is classified as a fingerprint difference, automatic repair is performed, followed by a second fingerprint verification. If successful, the migration is complete; otherwise, the shards are rolled back. If the difference is classified as a sampling difference, manual review is conducted directly.

[0118] This invention automatically categorizes differences and directly and automatically repairs fingerprint discrepancies, reducing repair time and requiring secondary verification to confirm the correctness of the automatic repair. For sampled differences, manual review is performed directly; that is, only complex anomalies are escalated to manual review, significantly reducing maintenance workload.

[0119] The embodiments of the present invention can also provide a continuous optimization mechanism, such as fingerprint strategy optimization. If the historical fingerprint difference rate is greater than a preset value, such as 0.01%, then field-level fingerprints are enabled; if the fingerprint generation time is greater than a preset time, such as 5 minutes, then the fingerprints are split into fragments.

[0120] For intelligent sampling models, the latest discrepancy data can be used to train the intelligent sampling model every week, and model retraining is triggered when the model AUC is less than 0.85.

[0121] As can be seen, in the above scheme, when performing data comparison, the source data cluster and the target data cluster are compared in a hierarchical and parallel manner, allowing multiple shards in each layer to perform data comparison tasks simultaneously and in parallel, thereby improving the data comparison speed and reducing the data comparison time. In addition, by binding different comparison strategies to each data layer, the data loading, transformation and storage process can be optimized, thereby improving the overall data processing efficiency.

[0122] The data verification method for big data cluster migration according to embodiments of the present invention has the following advantages:

[0123] High efficiency: Enables rapid comparison of petabyte-scale data, reducing processing time;

[0124] Accuracy: Ensures consistency between data content, structure, and business logic;

[0125] Flexibility: Supports both full migration and incremental migration scenarios.

[0126] Resource optimization: Reduce the consumption of computing and storage resources.

[0127] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0128] In one embodiment, a data verification device for big data cluster migration is provided, which corresponds one-to-one with the data verification method for big data cluster migration in the above embodiments. For example... Figure 10 As shown, the data verification device in this big data cluster migration includes a data acquisition module, a layering module, a comparison module, and a repair module. Detailed descriptions of each functional module are as follows:

[0129] The data acquisition module is used to acquire source data clusters and target data clusters;

[0130] The layering module is used to divide the source data cluster and the target data cluster into multiple data layers according to the standard data warehouse layering strategy, and bind different data comparison strategies to each data layer.

[0131] The comparison module is used to perform data fragmentation processing on each of the data layers and control each data layer to execute the data comparison task in parallel.

[0132] The repair module is used to record the difference data and perform data repair when a difference is detected in any data layer of the target data cluster and the source data cluster.

[0133] In an optional embodiment of the present invention, the layering module is further configured to divide the source data cluster and the target data cluster into a business data source layer, a data detail layer, a data summary layer, and a data indicator layer according to a standard data warehouse layering strategy.

[0134] The business data source layer is bound to an incremental fingerprint comparison strategy;

[0135] The data detail layer is bound to an intelligent sampling strategy;

[0136] The data aggregation layer is bound to key indicator fingerprints and sampling collaboration strategies;

[0137] The data metrics layer is bound to the golden metric fingerprint and the business-sensitive sample sampling strategy.

[0138] In an optional embodiment of the present invention, the comparison module is also used to divide the business data into shards according to the data volume and field attributes of the table; wherein, the table containing the LOB field is sharded separately and field-level fingerprinting is enabled;

[0139] The data detail layer is split according to the comparison table partition dimensions, which include time dimension, region dimension and business line dimension.

[0140] The data aggregation layer is allocated according to subject domains, and high-frequency indicators are split into independent shards; the subject domains include customers, transactions and / or products;

[0141] The data metrics layer is divided and categorized by report unit, with the gold index divided into separate segments.

[0142] In an optional embodiment of the present invention, the comparison module is further configured to verify each fragment using the comparison strategy bound to each data layer;

[0143] Business data source layer: Calculate incremental fingerprints for each data segment and perform fingerprint comparison. If the fingerprint difference exceeds a preset threshold, perform intelligent sampling for supplementary inspection. If intelligent sampling finds high-risk samples, perform full-field hash comparison on the high-risk samples.

[0144] Data detail layer: Intelligent sampling is performed based on each data shard using an intelligent sampling model; if an abnormal field is found, fingerprint supplementary inspection is performed on the abnormal field; if an abnormal fingerprint is found, deep verification of business rules is triggered.

[0145] Data aggregation layer: Calculates the digital fingerprint corresponding to each data segment. If there is a fingerprint difference, the corresponding data segment will be intelligently sampled. If there is a high-risk dimension in the sampling result, the association relationship will be verified.

[0146] Data indicator layer: Calculate the golden indicator fingerprint corresponding to the report data and perform golden fingerprint comparison. If a difference is found, perform sensitive field sampling. If a logical error is found, trigger the data summary layer backtracking.

[0147] In an optional embodiment of the present invention, the data verification device in the above-mentioned big data cluster migration further includes an optimization module, which is used to: set scheduling priorities for different comparison strategies for each data layer; and / or,

[0148] A priority queue is set up. If there are insufficient comparison resources, comparison tasks are executed according to the priority queue. The priority queue includes high priority, medium priority, and low priority. And / or,

[0149] Configure resource degradation strategies to automatically and dynamically adjust the comparison strategy based on the resource load of the target data cluster.

[0150] In an optional embodiment of the present invention, the data verification device in the above-mentioned big data cluster migration further includes a data preparation module, which is used to calculate the digital fingerprint corresponding to each data layer and generate a fingerprint baseline before data comparison.

[0151] The feature information of the data tables in the source data cluster is collected, and the feature information is processed by feature engineering to generate a sampling model feature library for use by the intelligent sampling model. The feature information includes data ontology features, business logic features, historical difference features and / or system metadata features.

[0152] In an optional embodiment of the present invention, the data verification device in the above-mentioned big data cluster migration further includes a metadata comparison module, which is used to collect the metadata corresponding to the source data cluster and the target data cluster respectively;

[0153] Based on the metadata corresponding to the source and target data clusters, check whether the table structures of the source and target data clusters are consistent. If they are consistent, continue the process; if they are inconsistent, pause the comparison task and perform a repair operation.

[0154] This invention provides a data verification device for big data cluster migration. When performing data comparison, it compares the source data cluster and the target data cluster in a hierarchical and parallel manner, allowing multiple shards in each layer to perform data comparison tasks simultaneously and in parallel, thereby improving the data comparison speed and reducing the data comparison time. In addition, by binding different comparison strategies to each data layer, the loading, transformation and storage processes of data can be optimized, thereby achieving the technical effect of improving the overall data processing efficiency.

[0155] Specific limitations regarding the data verification device in big data cluster migration can be found in the limitations of the data verification method in big data cluster migration described above, and will not be repeated here. Each module in the aforementioned data verification device for big data cluster migration can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0156] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the server-side functions or steps of a data verification method in a big data cluster migration.

[0157] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a data verification method in a big data cluster migration.

[0158] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0159] Obtain the source data cluster and the target data cluster;

[0160] According to the standard data warehouse layering strategy, the source data cluster and the target data cluster are divided into multiple data layers, and different data comparison strategies are bound to each data layer.

[0161] Perform data sharding on each of the data layers and control each data layer to execute the data comparison task in parallel.

[0162] If a difference is detected between any data layer of the target data cluster and the source data cluster, the difference data is recorded and data repair is performed.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0164] Obtain the source data cluster and the target data cluster;

[0165] According to the standard data warehouse layering strategy, the source data cluster and the target data cluster are divided into multiple data layers, and different data comparison strategies are bound to each data layer.

[0166] Perform data sharding on each of the data layers and control each data layer to execute the data comparison task in parallel.

[0167] If a difference is detected between any data layer of the target data cluster and the source data cluster, the difference data is recorded and data repair is performed.

[0168] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0169] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0170] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0171] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data verification method for big data cluster migration, characterized in that, include: Obtain the source data cluster and the target data cluster; According to the standard data warehouse layering strategy, the source data cluster and the target data cluster are divided into multiple data layers, and different data comparison strategies are bound to each data layer. Perform data sharding on each of the data layers and control each data layer to execute the data comparison task in parallel. If a difference is detected between any data layer of the target data cluster and the source data cluster, the difference data is recorded and data repair is performed.

2. The data verification method in big data cluster migration as described in claim 1, characterized in that, Based on the standard data warehouse layering strategy, the source data cluster and the target data cluster are divided into multiple data layers, and different data comparison strategies are bound to each data layer, including: According to the standard data warehouse layering strategy, the source data cluster and the target data cluster are respectively divided into the business data source layer, the data detail layer, the data summary layer, and the data indicator layer. The business data source layer is bound to an incremental fingerprint comparison strategy; The data detail layer is bound to an intelligent sampling strategy; The data aggregation layer is bound to key indicator fingerprints and sampling collaboration strategies; The data metrics layer is bound to the golden metric fingerprint and the business-sensitive sample sampling strategy.

3. The data verification method in big data cluster migration as described in claim 2, characterized in that, Data fragmentation processing for each of the data layers includes: Business data is sharded at the source layer based on the amount of data in the table and the field attributes; tables containing LOB fields are sharded separately and field-level fingerprints are enabled. The data detail layer is split according to the comparison table partition dimensions, which include time dimension, region dimension and business line dimension. The data aggregation layer is allocated according to subject domains, and high-frequency indicators are split into independent shards; the subject domains include customers, transactions and / or products; The data metrics layer is divided and categorized by report unit, with the gold index divided into separate segments.

4. The data verification method in big data cluster migration as described in claim 2, characterized in that, Controlling the parallel execution of data comparison tasks across data layers includes: verifying each fragment using the comparison strategy bound to each data layer; Business data source layer: Calculate incremental fingerprints for each data segment and perform fingerprint comparison. If the fingerprint difference exceeds a preset threshold, perform intelligent sampling for supplementary inspection. If intelligent sampling finds high-risk samples, perform full-field hash comparison on the high-risk samples. Data detail layer: Intelligent sampling is performed based on each data shard using an intelligent sampling model; if an abnormal field is found, fingerprint supplementary inspection is performed on the abnormal field; if an abnormal fingerprint is found, deep verification of business rules is triggered. Data aggregation layer: Calculates the digital fingerprint corresponding to each data segment. If there is a fingerprint difference, the corresponding data segment will be intelligently sampled. If there is a high-risk dimension in the sampling result, the association relationship will be verified. Data indicator layer: Calculate the golden indicator fingerprint corresponding to the report data and perform golden fingerprint comparison. If a difference is found, perform sensitive field sampling. If a logical error is found, trigger the data summary layer backtracking.

5. The data verification method in big data cluster migration as described in claim 2, characterized in that, The method further includes: Set different matching strategies for scheduling priorities for each data layer; and / or, A priority queue is set up. If there are insufficient comparison resources, comparison tasks are executed according to the priority queue. The priority queue includes high priority, medium priority, and low priority. And / or, Configure resource degradation strategies to automatically and dynamically adjust the comparison strategy based on the resource load of the target data cluster.

6. The data verification method in big data cluster migration as described in any one of claims 1-5, characterized in that, After obtaining the source data cluster and the target data cluster, the method further includes: Before data comparison, calculate the digital fingerprint corresponding to each data layer and generate a fingerprint baseline; The feature information of the data tables in the source data cluster is collected, and the feature information is processed by feature engineering to generate a sampling model feature library for use by the intelligent sampling model. The feature information includes data ontology features, business logic features, historical difference features and / or system metadata features.

7. The data verification method in big data cluster migration as described in claim 6, characterized in that, The method further includes: Collect metadata corresponding to the source data cluster and the target data cluster respectively; Based on the metadata corresponding to the source and target data clusters, check whether the table structures of the source and target data clusters are consistent. If they are consistent, continue the process; if they are inconsistent, pause the comparison task and perform a repair operation.

8. A data verification method and apparatus for big data cluster migration, characterized in that, include: The data acquisition module is used to acquire source data clusters and target data clusters; The layering module is used to divide the source data cluster and the target data cluster into multiple data layers according to the standard data warehouse layering strategy, and bind different data comparison strategies to each data layer. The comparison module is used to perform data fragmentation processing on each of the data layers and control each data layer to execute the data comparison task in parallel. The repair module is used to record the difference data and perform data repair when a difference is detected in any data layer of the target data cluster and the source data cluster.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data verification method in big data cluster migration as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data verification method in big data cluster migration as described in any one of claims 1 to 7.

Images