A data consistency checking method and system based on efficient data sampling

Abstract

The application discloses a data consistency checking method and system based on efficient data sampling, which extracts representative sub-samples from large-scale data sets through hierarchical sampling and optimal probability calculation, avoids large resource and time consumption required by full-scale data comparison under the premise of ensuring checking accuracy. In addition, the data is divided into different layers according to the data characteristics, sampling is performed in each layer, and the optimal sampling probability is recalculated to enhance the representativeness of the sampling data in the overall data. The hierarchical dimension intelligent recommendation engine solves the problem of poor adaptability of the hierarchical dimension in the prior art, and further improves the sampling representativeness and system adaptability. After data sampling comparison, the inconsistent data rows, columns and difference information of related fields between the master and standby libraries are recorded and classified and displayed in detail, so that the user is provided with more intuitive results, the user can quickly identify the data inconsistency problem, and further correction measures are taken according to the difference report and repair suggestion generated by the system.

Description

Technical Field

[0001] This invention relates to a data consistency verification method and system based on efficient data sampling, belonging to the field of database management. Background Technology

[0002] As business scope expands, large-scale enterprises in sectors like finance and telecommunications, due to their large data volumes, often choose a master-slave database architecture to ensure high availability of core business operations. This architecture provides functions such as master-slave database data synchronization, data backup and recovery, read-write separation, and high-availability failover. However, master database failures, unpredictable operations on the master database, or database migration operations can lead to risks such as accidental log loss or data inconsistency between the master and slave databases. If a master-slave switch occurs, the slave database becoming the new master will result in data inconsistency, causing business logic chaos. To avoid these situations, enterprises generally employ various methods to verify database data consistency. Theoretically, full data comparison provides the most comprehensive and accurate data consistency verification; however, full comparison requires relatively high time and computing resources, often resulting in significant performance degradation in practice. Furthermore, in high-concurrency production environments, the execution time of data consistency checks can be lengthy, impacting normal database operation. Most comparison tools require manual triggering, and operations personnel need to perform checks periodically. In environments with frequent changes, oversights may occur, leading to potential data consistency issues. In addition, many tools require command-line operation throughout and involve numerous configuration steps. For non-professional developers, this requires a significant amount of time to learn and adapt, which increases the cost for businesses.

[0003] As databases grow in size, inconsistencies may arise between them due to various factors. Data consistency verification compares the data in two databases to check for consistency and generates reports on any discrepancies found. To ensure business system stability and data consistency, enterprises need to implement additional monitoring and synchronization measures to address potential data inconsistencies in different scenarios. Attached Figure Description

[0004] Figure 1 This is a system flowchart. Summary of the Invention

[0005] To address the inefficiency in processing massive or complex structured data, this invention proposes a data consistency verification system based on efficient data sampling. By selecting a representative subset of data from a large-scale dataset, the system ensures the reliability of verification results and significantly reduces system overhead. The system includes:

[0006] (1) Initialization selection module. In this module, the user selects a specific database object type through the interface and extracts the metadata in the database, including structural information such as tables, columns, indexes, and stored procedures, to facilitate subsequent comparison and use.

[0007] (2) Object comparison module. In this module, the existence of objects in the master database and the standby database is verified, the differences in table structure and columns are compared, index information is extracted and the names and order are checked, the definitions and parameters of stored procedures and triggers are compared, and all differences are recorded.

[0008] (3) Data comparison module, which adjusts the sampling probability according to the randomness and representativeness of the sample to optimize the representativeness and test accuracy of the sample;

[0009] (4) Difference Display Module: This module displays the data comparison results to the user through a detailed report.

[0010] Specifically, the data comparison module performs data comparisons using the following steps:

[0011] 1) Data sampling: Extract a set of valid subsamples from the dataset, perform statistical analysis on only the subsamples to obtain inference results, and assign different sampling probabilities to different data points.

[0012] (a) Data Stratification: Data is segmented based on the specific characteristics of data objects in the database (such as timestamps, regions, categories, user levels, etc.) to form different strata. The total sample size of the database is... The data is divided into K layers, and the dataset for each layer is denoted as follows: The total number of samples contained in each layer is , k=1,2,…,K.

[0013] (b) Stratified sampling and parameter estimation: First, given the initial sample size , The sampling probability for each layer is , And retrieve data from the master database. As a basis for comparison, extract from the backup database As a comparison target, each layer of the dataset is constructed .

[0014] Parameter estimation is performed based on the initial sampled data for each layer. For each layer's dataset, based on... and Constructor function Estimate model parameters ,in This refers to the data value of the i-th sample in the k-th layer extracted from the main database. The comparison target value extracted from the k-th layer of the backup database. The initial probability of each sample in the stratum. This represents the estimator of the sample from the k-th stratum; *0 indicates the initial sampling; (Note: In sampling theory, the subscript or superscript "0" is usually used to represent initial or baseline parameters, such as the null hypothesis H0 and initial parameters θ0 in statistics; in this invention, *0 is used as a unified superscript to represent quantities related to the initial sampling stage, to distinguish them from the corresponding quantities calculated after optimal probability reestimation); calculate the estimator of the sampled data rows. .

[0015] (c) Calculate the optimal sampling probability: Redefine the sampling probability based on the sampled data, and calculate the optimal sampling probability for each layer. ,in This represents the data values ​​of the overall dataset at each level. To compare the data values ​​of each layer of the overall dataset in the database, Predict the probability for the sampled sample in the k-th layer. For each layer of the overall dataset, the predicted probability is... For each layer's weight matrix, .

[0016] (d) Sampling implementation: based on Determine the sampling location for the sample, and perform stratified sampling on the main database and backup database to ensure that the stratified structure is followed and that the sampling is representative;

[0017] 2) Data comparison: Compare the data sampled from the primary database and the standby database line by line to check whether the sampled data is consistent between the primary and standby databases, and identify and record the inconsistent data rows and the reasons for the differences.

[0018] When stratifying data, an intelligent recommendation engine for stratified dimensions is adopted. Based on the data distribution characteristics of the database, the optimal combination of stratified dimensions is automatically selected through machine learning algorithms, and the weight ratio of each stratum is dynamically adjusted. At the same time, the sampling error of each stratum is monitored in real time. When the error exceeds the preset threshold, the stratified dimensions are automatically reselected or the number of strata is adjusted.

[0019] Based on the above system, this invention proposes a data consistency verification method based on efficient data sampling, which includes the following steps:

[0020] (1) Initialization selection: Users select a specific database object type through the interface to extract metadata from the database, including structural information such as tables, columns, indexes, and stored procedures, which is convenient for subsequent comparison and use;

[0021] (2) Object comparison: By verifying the existence of objects in the primary database and the backup database, comparing the differences in table structure and columns, extracting index information and checking names and order, comparing the definitions and parameters of stored procedures and triggers, and recording all differences;

[0022] (3) Data comparison: Adjust the sampling probability according to the randomness and representativeness of the sample to optimize the representativeness and test accuracy of the sample;

[0023] (4) Difference display: The data comparison results are displayed to the user through a detailed report.

[0024] Specifically, the following steps are used when comparing data:

[0025] 1) Data sampling: Extract a set of valid subsamples from the dataset, perform statistical analysis on only the subsamples to obtain inference results, and assign different sampling probabilities to different data points.

[0026] (a) Data Stratification: Data is segmented based on the specific characteristics of data objects in the database (such as timestamps, regions, categories, user levels, etc.) to form different strata. The total sample size of the database is... The data is divided into K layers, and the dataset for each layer is denoted as follows: The total number of samples contained in each layer is , k=1,2,…,K.

[0027] (b) Stratified sampling and parameter estimation: First, given the initial sample size , The sampling probability for each layer is , And retrieve data from the master database. As a basis for comparison, extract from the backup database As a comparison target, each layer of the dataset is constructed .

[0028] Parameter estimation is performed based on the initial sampled data for each layer. For each layer's dataset, based on... and Constructor function Estimate model parameters ,in This refers to the data value of the i-th sample in the k-th layer extracted from the main database. The comparison target value extracted from the k-th layer of the backup database. The initial probability of each sample in the stratum. Let *0 be the estimator of the sampled data at stratum k; where the superscript *0 indicates the initial sampling (in sampling theory, the subscript or superscript "0" is usually used to represent the initial or baseline parameters; in this invention, *0 is used as a unified superscript to represent quantities related to the initial sampling stage, to distinguish them from the corresponding quantities after optimal probability reestimation); calculate the estimator of the sampled data rows. .

[0029] (c) Calculate the optimal sampling probability: Redefine the sampling probability based on the sampled data, and calculate the optimal sampling probability for each layer. ,in This represents the data values ​​of the overall dataset at each level. To compare the data values ​​of each layer of the overall dataset in the database, Predict the probability for the sampled sample in the k-th layer. For each layer of the overall dataset, the predicted probability is... For each layer's weight matrix, .

[0030] (d) Sampling implementation: based on Determine the sampling location for the sample, and perform stratified sampling on the main database and backup database to ensure that the stratified structure is followed and that the sampling is representative;

[0031] 2) Data comparison: Compare the data sampled from the primary database and the standby database line by line to check whether the sampled data is consistent between the primary and standby databases, and identify and record the inconsistent data rows and the reasons for the differences.

[0032] When stratifying data, an intelligent recommendation engine for stratified dimensions is adopted. Based on the data distribution characteristics of the database, the optimal combination of stratified dimensions is automatically selected through machine learning algorithms, and the weight ratio of each stratum is dynamically adjusted. At the same time, the sampling error of each stratum is monitored in real time. When the error exceeds the preset threshold, the stratified dimensions are automatically reselected or the number of strata is adjusted.

[0033] This invention extracts metadata from the master and slave databases through an initialization selection module, performs consistency verification on complex objects such as table structures, indexes, and stored procedures through an object comparison module, and employs a closed-loop strategy of data stratification, parameter estimation, optimal sampling probability calculation, and stratified sampling through a data comparison module to ensure verification accuracy and significantly reduce system overhead. The difference display module outputs a categorized difference report and repair suggestions through a visual interface. This invention achieves full-dimensional automated verification from object structure to data values, adapts to large-scale, complex structured data scenarios, significantly improves verification efficiency and ease of use, and can be widely applied in industries such as finance, e-commerce, and government that rely on master-slave database architectures.

[0034] This invention employs stratified sampling and optimal probability calculation to extract representative subsamples from large-scale datasets. This avoids the significant resource and time overhead required for full-scale data comparison while ensuring verification accuracy. Furthermore, by dividing the data into different strata based on data characteristics, sampling is performed within each stratum, and the optimal sampling probability is recalculated. This enhances the representativeness of the sampled data within the overall dataset. The stratified dimension intelligent recommendation engine addresses the issues of manual configuration and poor adaptability in existing technologies, further improving sampling representativeness and system adaptability. After data sampling and comparison, detailed records and categorized displays of inconsistencies in rows, columns, and related fields between the primary and backup databases provide users with more intuitive results. This facilitates quick identification of data inconsistencies and allows users to take corrective measures based on the system-generated discrepancy reports and repair suggestions. Detailed Implementation

[0035] To address the challenges of poor performance in large-scale data volume verification, difficulties in handling data types and complex structures, and challenges in verifying partial data in database consistency checks, this application employs a sampling comparison technique during the data comparison process. This reduces the computational complexity and time consumption of large-scale full-volume comparisons. The system flowchart is shown below. Figure 1 .

[0036] 1. Initialize the selection module

[0037] Users select a specific database object type through the interface. The system connects to the database based on the connection information of the selected primary and standby databases, and extracts the metadata of the primary and standby databases, including the structural information of tables, columns, indexes, stored procedures, etc., for consistency verification.

[0038] 2. Object Comparison Module

[0039] (1) Object existence verification

[0040] The system iterates through the user-selected objects, checking their existence in the primary and standby databases. If an object is missing, it is recorded and marked as "missing" in the report, along with detailed information about the missing object, including its name, type, and related information.

[0041] (2) Comparison of table structures

[0042] Check if the number of columns in the database is consistent, and then perform the following comparisons for each column: check if the column names are consistent and mark the different columns; compare data types to ensure type matching and record type differences; check constraints, compare primary keys, foreign keys, uniqueness, and other constraints, and record differences.

[0043] (3) Index comparison

[0044] Extract detailed information for all indexes in the primary and standby databases, compare index names, types, associated columns, and order, and record differences in names and column order.

[0045] (4) Trigger comparison

[0046] Extract the definitions and parameters of stored procedures and triggers, and compare the names, definition bodies, and parameter lists.

[0047] 3. Data Comparison Module

[0048] The system determines the sampling samples from the primary and backup databases by initializing the table structure, columns, and index information obtained from the selection module and object comparison module. The object comparison module ensures structural consistency between the primary and backup databases, enabling the system to select a more suitable sample set for comparative analysis based on known table structures and data types during the data comparison phase. During sampling, the system compares data based on consistent table structures and column data between the primary and backup databases to ensure the representativeness of the data consistency check. The data comparison process involves the following steps:

[0049] (1) Data sampling

[0050] A valid subsample is extracted from the dataset, statistical analysis is performed on the subsample to obtain inference results, and different sampling probabilities are assigned to different data points. Specifically:

[0051] a. Data layering

[0052] The data in the database is segmented based on specific characteristics of the data objects (such as timestamps, regions, categories, user levels, etc.), forming different strata. The total sample size of the database is [missing information]. The data is divided into K layers, and the datasets in each layer are labeled as follows: The total number of samples contained in each layer is .

[0053] When data is stratified, a stratified dimension intelligent recommendation engine is used. Based on the data distribution characteristics of the database (such as field repetition rate, data update frequency, field importance weight, etc.), the optimal combination of stratification dimensions (such as timestamp + user level, region + data category) is automatically selected through machine learning algorithms (such as decision tree model), and the weight ratio of each stratum is dynamically adjusted. At the same time, the sampling error of each stratum is monitored in real time. When the error exceeds the preset threshold, the stratification dimension reselection or the number of strata is automatically triggered.

[0054] b. Stratified Sampling and Parameter Estimation

[0055] In each stratum, to ensure a representative sample size, data will be sampled according to a set sampling probability, as follows:

[0056] First, an initial sampling number is given for each layer. , Set the sampling probability for each layer Extract data from the master database As a basis for comparison, relevant data was extracted from the backup database. As a comparison target, each layer of the dataset is composed of .

[0057] Parameter estimation is performed based on the initial sampled data for each layer. For each layer's dataset, based on the main database data... and backup database data Constructor function Estimate model parameters This represents the data value of the i-th sample in the k-th layer extracted from the main database. This represents the comparison target value extracted from the k-th layer of the backup database. This represents the initial probability of each sample in the sampling layer. is the estimate of the sample from the k-th stratum; where the superscript *0 indicates the initial sampling (in sampling theory, the subscript or superscript "0" is usually used to represent the initial or baseline parameters. In this invention, *0 is used as a unified superscript to represent the quantity related to the initial sampling stage, so as to distinguish it from the corresponding quantity after the optimal probability re-estimation).

[0058] Calculate the estimator of the sampled data rows. The process of solving this parameter plays a crucial role in complex, large-scale data environments, effectively estimating the data distribution and thus providing a more accurate basis for subsequent sampling operations.

[0059] c. Calculate the optimal sampling probability

[0060] Redefine sampling probabilities based on the sampled data to improve sampling efficiency and accuracy. Calculate the optimal sampling probability for each stratum. ,in This represents the data values ​​of the overall dataset at each level. To compare the data values ​​of each layer of the overall dataset in the database, Predict the probability for the sampled sample in the k-th layer. For each layer of the overall dataset, the predicted probability is... For each layer's weight matrix, .

[0061] The optimal sampling probability allows for greater focus on data rows with significant differences during optimal sampling, thereby improving the accuracy of data sampling.

[0062] d. Sampling implementation

[0063] Based on the optimal sampling probability calculated for each layer The sampling location of the sample is determined, and stratified sampling is performed on the data in the main database and the backup database to ensure that the sampling is representative and follows the stratified structure.

[0064] (2) Data comparison

[0065] Ensure data row and column alignment. Based on the previous table structure comparison results, ensure that the fields and rows in the primary and backup databases correspond and match. Then, for each field... Perform value The system performs a step-by-step comparison, handling fields of different types such as numeric, character, and date data to ensure consistency between the primary and standby databases. If discrepancies are found, the system records detailed information about the differences, including table names, column names, primary keys or unique identifiers, and the specific value differences between the primary and standby databases.

[0066] 4. Difference Display Module

[0067] After completing the data comparison, the system first extracts inconsistent data rows and related information from the primary and standby databases, including inconsistent table names, column names, primary keys or unique identifiers, and specific field value differences between the primary and standby databases. The system then categorizes and summarizes this data according to different dimensions (such as table name, field name, and difference type).

[0068] Users can click on specific tables or fields through the interface to view their detailed difference records and choose to generate a difference report, which includes detailed difference records, difference statistics, and remediation suggestions.

[0069] Example 1

[0070] Users select two database objects on the interface to perform data consistency verification, such as the primary database MainDB and the standby database BackupDB. The system then connects to the database based on the connection information and extracts the data.

[0071] Object comparison module: First, it checks the existence of objects in the primary database MainDB and the standby database BackupDB. Then, it compares the table structures. Assuming the Users table structures in the primary database MainDB and the standby database BackupDB are as follows:

[0072] id INT INT name VARCHAR(100) VARCHAR(100) email VARCHAR(100) VARCHAR(150) created_at TIMESTAMP TIMESTAMP

[0073] Comparison results: The data types of the email column are inconsistent, resulting in discrepancies in the records.

[0074] Index comparison: Assume the primary database has the index idx_email, but the standby database lacks it.

[0075] Trigger comparison: The primary database has the before_insert_trigger trigger, while the standby database lacks it.

[0076] Data comparison module: Assuming the MainDB.Users table has The row data is divided into four levels based on the table structure type, and the sampling quantity for the first level is set. The sampling probability is 0.01.

[0077] Suppose the data in the sampled data row is as follows:

[0078] 1 Alice alice@example.com 2024-10-01 12:00:00 2 Bob bob@example.com 2024-10-03 12:00:00 3 Charlie charlie@example.com 2024-10-03 12:00:00

[0079] Based on the initial sampling data for each layer, parameter estimation is performed. For each layer's dataset, a function is constructed using the primary database data and the backup database data. The parameters of the estimated model are calculated using the following formula:

[0080] .

[0081] Assuming that the estimated number of sampled data rows is 25 during the solution process, this provides a more accurate basis for the next sampling operation.

[0082] When data is stratified, a stratified dimension intelligent recommendation engine is used. Based on the data distribution characteristics of the database (such as field repetition rate, data update frequency, field importance weight, etc.), the optimal combination of stratification dimensions (such as timestamp + user level, region + data category) is automatically selected through machine learning algorithms (such as decision tree model), and the weight ratio of each stratum is dynamically adjusted. At the same time, the sampling error of each stratum is monitored in real time. When the error exceeds the preset threshold, the stratification dimension reselection or the number of strata is automatically triggered.

[0083] Calculate the optimal sampling probability: Assuming the sampling of layer 1, calculate the optimal sampling probability for each row of data. .

[0084] Based on the optimal sampling probability calculated for each layer The sampling location of the sample is determined, and stratified sampling is performed on the data in the main database and the backup database to ensure that the sampling is representative and follows the stratified structure.

[0085] Data comparison:

[0086] Ensure that the rows and columns of the data are aligned. Based on the previous table structure comparison results, ensure that the fields and rows in the primary and backup databases can correspond and match.

[0087] Then perform the difference output, with the output format as follows: "Table name: Users, Column name: email, Main database value: " bob@ example.com Backup value: None;

[0088] The difference display module: After completing the data comparison, the system extracts the inconsistent data rows and related information, outputting information such as inconsistent table names, column names, primary keys, and specific value differences. Users can view the difference records through the interface and generate a difference report.

[0089] The units, devices, or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the devices or units shown or discussed can be through some interfaces, and the indirect coupling or communication connection between the devices or units can be electrical, mechanical, or other forms.

[0090] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0091] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0092] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0093] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0094] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data consistency verification system based on efficient data sampling, the system comprising an initialization selection module, an object comparison module, a data comparison module, and a difference display module, characterized in that: The initialization selection module allows users to select specific database object types through the interface and extract metadata from the database, including structural information such as tables, columns, indexes, and stored procedures, for easy comparison later. The object comparison module verifies the existence of objects in the primary and backup databases, compares the differences in table structure and columns, extracts index information and checks names and order, compares the definitions and parameters of stored procedures and triggers, and records all differences. The data comparison module adjusts the sampling probability based on the randomness and representativeness of the sample, thereby optimizing the representativeness of the sample and the accuracy of the test. Difference Display Module: This module presents the data comparison results to users through detailed reports.

2. The data consistency verification system based on efficient data sampling as described in claim 1, characterized in that: The data comparison module performs data comparisons using the following steps: (1) Data sampling: Extract a set of valid subsamples from the dataset, perform statistical analysis on only the subsamples to obtain inference results, and assign different sampling probabilities to different data points. (a) Data Stratification: Data is segmented based on the specific characteristics of data objects in the database (such as timestamps, regions, categories, user levels, etc.) to form different strata. The total sample size of the database is... The data is divided into K layers, and the dataset for each layer is denoted as follows: The total number of samples contained in each layer is , k=1,2,…,K. (b) Stratified sampling and parameter estimation: First, given the initial sample size , The sampling probability for each layer is , And retrieve data from the master database. As a basis for comparison, extract from the backup database As a comparison target, each layer of the dataset is constructed . Parameter estimation is performed based on the initial sampled data for each layer. For each layer's dataset, based on... and Constructor function Estimate model parameters ,in This refers to the data value of the i-th sample in the k-th layer extracted from the main database. The comparison target value extracted from the k-th layer of the backup database. The initial probability of each sample in the stratum. It is an estimate of the sample from the k-th stratum. Calculate the estimator of the sampled data rows. *0 indicates initial sampling. (c) Calculate the optimal sampling probability: Redefine the sampling probability based on the sampled data, and calculate the optimal sampling probability for each layer. ,in This represents the data values ​​of the overall dataset at each level. To compare the data values ​​of each layer of the overall dataset in the database, Predict the probability for the sampled sample in the k-th layer. For each layer of the overall dataset, the predicted probability is... For each layer's weight matrix, . (d) Sampling implementation: based on Determine the sampling location for the sample, and perform stratified sampling on the main database and backup database to ensure that the stratified structure is followed and that the sampling is representative; (2) Data comparison: Compare the data sampled from the master database and the backup database line by line, check whether the sampled data is consistent between the master and backup databases, identify and record inconsistent data lines and reasons for the differences.

3. The data consistency verification system based on efficient data sampling as described in claim 2, characterized in that: When stratifying data, an intelligent recommendation engine for stratified dimensions is adopted. Based on the data distribution characteristics of the database, the optimal combination of stratified dimensions is automatically selected through machine learning algorithms, and the weight ratio of each stratum is dynamically adjusted. At the same time, the sampling error of each stratum is monitored in real time. When the error exceeds the preset threshold, the stratified dimensions are automatically reselected or the number of strata is adjusted.

4. A data consistency verification method based on efficient data sampling, comprising four steps: initialization selection, object comparison, data comparison, and difference display, characterized in that: (1) Initialization selection: Users select a specific database object type through the interface to extract metadata from the database, including structural information such as tables, columns, indexes, and stored procedures, which is convenient for subsequent comparison and use; (2) Object comparison: By verifying the existence of objects in the primary database and the backup database, comparing the differences in table structure and columns, extracting index information and checking names and order, comparing the definitions and parameters of stored procedures and triggers, and recording all differences; (3) Data comparison: Adjust the sampling probability according to the randomness and representativeness of the sample to optimize the representativeness and test accuracy of the sample; (4) Difference display: The data comparison results are displayed to the user through a detailed report.

5. The data consistency verification method based on efficient data sampling as described in claim 4, characterized in that: When comparing data, the following steps are used: (1) Data sampling: Extract a set of valid subsamples from the dataset, perform statistical analysis on only the subsamples to obtain inference results, and assign different sampling probabilities to different data points. (a) Data Stratification: Data is segmented based on the specific characteristics of data objects in the database (such as timestamps, regions, categories, user levels, etc.) to form different strata. The total sample size of the database is... The data is divided into K layers, and the dataset for each layer is denoted as follows: The total number of samples contained in each layer is , k=1,2,…,K. (b) Stratified sampling and parameter estimation: First, given the initial sample size , The sampling probability for each layer is , And retrieve data from the master database. As a basis for comparison, extract from the backup database As a comparison target, each layer of the dataset is constructed . Parameter estimation is performed based on the initial sampled data for each layer. For each layer's dataset, based on... and Constructor function Estimate model parameters ,in This refers to the data value of the i-th sample in the k-th layer extracted from the main database. The comparison target value extracted from the k-th layer of the backup database. The initial probability of each sample in the stratum. It is an estimate of the sample from the k-th stratum. Calculate the estimator of the sampled data rows. *0 indicates initial sampling. (c) Calculate the optimal sampling probability: Redefine the sampling probability based on the sampled data, and calculate the optimal sampling probability for each layer. ,in This represents the data values ​​of the overall dataset at each level. To compare the data values ​​of each layer of the overall dataset in the database, Predict the probability for the sampled sample in the k-th layer. For each layer of the overall dataset, the predicted probability is... For each layer's weight matrix, . (d) Sampling implementation: based on Determine the sampling location for the sample, and perform stratified sampling on the main database and backup database to ensure that the stratified structure is followed and that the sampling is representative; (2) Data comparison: Compare the data sampled from the master database and the backup database line by line, check whether the sampled data is consistent between the master and backup databases, identify and record inconsistent data lines and reasons for the differences.

6. The data consistency verification method based on efficient data sampling as described in claim 5, characterized in that: When stratifying data, an intelligent recommendation engine for stratified dimensions is adopted. Based on the data distribution characteristics of the database, the optimal combination of stratified dimensions is automatically selected through machine learning algorithms, and the weight ratio of each stratum is dynamically adjusted. At the same time, the sampling error of each stratum is monitored in real time. When the error exceeds the preset threshold, the stratified dimensions are automatically reselected or the number of strata is adjusted.

Images