Data comparison method and device, electronic equipment and storage medium

Abstract

The invention discloses a data comparison method and device, electronic equipment and a storage medium, belongs to the technical field of data processing, and is used for improving the accuracy of data comparison. The method comprises the following steps: acquiring to-be-compared first data from a plurality of databases; sampling the first data through a plurality of sampling strategies to obtain second data corresponding to each sampling strategy; according to the second data, determining a target sampling strategy meeting a predetermined condition from a plurality of sampling strategies; and comparing the second data corresponding to the target sampling strategy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a data comparison method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In related technologies, data is extracted from a database based on a specific query statement or a specific field for data comparison. This data comparison method cannot extract representative sample data for subsequent data comparison when facing uneven data distribution, excessive data volume, database quality differences (different data formats, encoding methods, missing value processing, etc.), and performance bottlenecks, resulting in low data comparison accuracy. SUMMARY

[0003] The embodiments of the application provide a data comparison method, device, electronic equipment and storage medium, which can solve the problem of low data comparison accuracy.

[0004] In a first aspect, the embodiments of the application provide a data comparison method, which comprises: obtaining first data to be compared from a plurality of databases; sampling the first data through a plurality of sampling strategies to obtain second data corresponding to each sampling strategy; determining a target sampling strategy meeting a predetermined condition from the plurality of sampling strategies according to the second data; and comparing the second data corresponding to the target sampling strategy.

[0005] In a second aspect, the embodiments of the application provide a data comparison device, which comprises: an obtaining module configured to obtain first data to be compared from a plurality of databases; a sampling module configured to sample the first data through a plurality of sampling strategies to obtain second data corresponding to each sampling strategy; a determining module configured to determine a target sampling strategy meeting a predetermined condition from the plurality of sampling strategies according to the second data; and a comparison module configured to compare the second data corresponding to the target sampling strategy.

[0006] In a third aspect, the embodiments of the application provide electronic equipment, which comprises a processor, a memory, and a program or instructions stored on the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the method of the first aspect.

[0007] In a fourth aspect, the embodiments of the application provide a readable storage medium, and the readable storage medium stores a program or instructions, and the program or instructions are executed by the processor to implement the steps of the method of the first aspect.

[0008] In this embodiment of the application, by obtaining first data to be compared from multiple databases; sampling the first data using multiple sampling strategies to obtain second data corresponding to each sampling strategy; determining a target sampling strategy that meets predetermined conditions from multiple sampling strategies based on the second data; and comparing the second data corresponding to the target sampling strategy, the accuracy of data comparison can be improved. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating a data comparison method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a data comparison device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0013] The data comparison method, apparatus, electronic device, and storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0014] Figure 1 This application illustrates a data comparison method provided by an embodiment of the present application. This method can be executed by a data comparison system and includes the following steps: Step S101: Obtain the first data to be compared from multiple databases.

[0015] During data comparison, the data to be compared may be stored in different types of databases, such as GaussDB and PostgreSQL. Therefore, this application embodiment provides multi-database (data source) support, supports dynamic switching between multiple databases, and has good scalability and flexibility. In this application embodiment, the information of each database can be configured in the database configuration file. This information typically includes the database type, URL, username, password, etc. For example, based on open-source frameworks such as Spring Boot and XXL-Job, the information of each database can be configured in the application.properties or application.yml file.

[0016] In this embodiment, a dynamic database manager is constructed, which adopts Spring's AbstractRoutingDataSource class. By implementing a dynamic routing mechanism in DataSourceRouting, it can intelligently switch between multiple databases according to the needs of the current data comparison task and the source of the data comparison object.

[0017] Built on open-source frameworks such as Spring Boot and XXL-Job, it features excellent scalability and flexibility. Databases can be easily added or modified, scheduled task configurations adjusted, data processing logic optimized, and data comparison discrepancies located, adapting to data processing scenarios of varying scales and complexities.

[0018] In one embodiment, before obtaining the first data to be compared from multiple databases, the method further includes: obtaining pre-created target connections to multiple databases from a connection pool; and connecting to the multiple databases through the target connections.

[0019] In this implementation, a suitable database connection can be automatically selected based on the business scenario. Dynamic database routing and scheduling are achieved by setting a context identifier. In this embodiment, a connection pool is pre-created for each database, and connections to the database are pre-created within each pool. Each database connection pool is managed by Spring's HikariCP, ensuring performance optimization under high concurrency. Therefore, in this embodiment, pre-created target connections to multiple databases can be obtained from the connection pool, and multiple databases can be connected through these target connections. By using HikariCP to manage the database connection pool, in the same data comparison scenario, each thread processes the same database. A connection can be established at the beginning of each thread and reused throughout the thread's execution, avoiding the overhead of multiple database connections and disconnections, thereby improving the overall program performance.

[0020] In this embodiment, after configuring multiple databases, the first data to be compared can be efficiently read from multiple databases by connecting to the databases. Combined with a database manager, the first data to be compared can be easily read from multiple databases.

[0021] When performing data comparison tasks, the system can automatically adjust database connections to ensure efficient and accurate data reading. For example, when reading the first data to be compared from database A (such as GaussDB) and database B (such as PostgreSQL), the system can automatically select the appropriate database, avoiding the tedious manual configuration.

[0022] In this embodiment, the scalability and high availability of the database are improved through Spring Boot's auto-configuration and HikariCP database connection pool management. When more databases or more complex data processing are required, only the database information needs to be added through the configuration file and the task execution logic needs to be adapted, without refactoring the architecture of the data comparison system.

[0023] Meanwhile, connection pooling technology ensures efficient management of database connections, avoiding frequent connection creation and disconnection, and greatly improving the throughput and response speed of the data comparison system. Spring Boot's integration testing capabilities ensure smooth switching and efficient interaction between different databases, guaranteeing the stability of the data comparison system.

[0024] The multi-database configuration mechanism in this embodiment can dynamically select the most suitable database for the current comparison task, improving the efficiency of data reading and comparison. It not only enhances the flexibility and scalability of database management but also ensures high efficiency and stable performance under high concurrency and large-scale data processing through HikariCP connection pooling and dynamic routing technology.

[0025] In one embodiment, obtaining the first data to be compared from multiple databases includes: obtaining the first data from multiple databases according to a configuration table, wherein the configuration table includes database identifiers of multiple databases, data tables corresponding to the first data, key fields of the first data, and filtering conditions for the first data.

[0026] In this embodiment, a configuration table mechanism is added, predefining databases and comparison rules in the `fullamountthancfg` database table. This configuration table can include database identifiers for multiple databases requiring data comparison, the data table corresponding to the first data, key fields of the first data (used for the first data and key fields for performing the comparison), filtering conditions for the first data (e.g., time range, numerical range, etc., used to filter the data to be compared), and sorting fields for the first data (used to ensure data consistency and order), aiming to provide precise execution guidance for the data comparison task. In this embodiment, the extraction of the first data is closely integrated with the configuration table, enabling the reading of the first data to be compared from different databases based on the database information in the configuration table.

[0027] In this embodiment, information about the first data to be compared can be extracted from the configuration table, and SQL query statements can be dynamically generated for data reading. By integrating JdbcTemplate and DataSourceContextHolder into the Spring Boot framework, the database can be automatically switched when each comparison task is executed, and a matching database query can be executed to extract the first data to be compared from the database.

[0028] For example, in financial applications, a data comparison system may need to extract transaction data from two different databases. JdbcTemplate will automatically switch to the database from which the comparison data needs to be extracted based on the configuration table and execute the query. Then, the extracted data will be loaded into memory for subsequent analysis.

[0029] In this embodiment of the application, a batch processing mechanism is provided to handle large amounts of first data to be compared. During the extraction of the first data, the first data to be compared can be divided into batches, and the processing of each batch of first data is optimized through multi-threading, thereby avoiding performance bottlenecks caused by insufficient memory or prolonged database connection occupation when processing extremely large amounts of data.

[0030] In this embodiment, the extracted first data can also be verified using a built-in verification module to ensure that the input first data meets the requirements for comparison and analysis. Especially in industries such as finance and healthcare, the accuracy and completeness of data are crucial; therefore, the system performs data quality verification on the extracted first data to detect issues such as missing data, duplication, and format errors.

[0031] In this embodiment of the application, the built-in verification module can automatically identify and report quality problems of the first data, such as missing field values ​​or mismatched data formats, and can record erroneous data to a dedicated log file for subsequent manual intervention or further analysis.

[0032] This application embodiment achieves efficient extraction and processing of comparison data, automatically reading information from the configuration table and ensuring efficient execution under large data volumes through multi-threaded batch processing and memory management mechanisms. Simultaneously, data quality verification and preprocessing guarantee the accuracy and consistency of the extracted initial data, laying a solid foundation for subsequent comparison and analysis tasks. This application embodiment improves the automation, flexibility, and scalability of the data extraction process while maintaining performance.

[0033] Step S102: Sample the first data using multiple sampling strategies to obtain the second data corresponding to each sampling strategy.

[0034] In this embodiment, sampling is a crucial step in the data comparison process. It aims to accurately extract first data using various sampling strategies to provide representative second data for subsequent data comparison and analysis. In this embodiment, the first data can be sampled using multiple sampling strategies to obtain second data corresponding to each strategy.

[0035] In one embodiment, the multiple sampling strategies include at least: random sampling, stratified sampling, systematic sampling, and conditional sampling. This application proposes multiple sampling strategies, such as random sampling, stratified sampling, systematic sampling, and conditional sampling, which can automatically select the most suitable sampling method for different data characteristics and analysis objectives. Through flexible combinations of sampling strategies, complex situations such as uneven data distribution, excessively large data volumes, or missing values ​​can be effectively addressed, ensuring that the samples are sufficiently representative, thereby improving the accuracy of subsequent data comparison and analysis.

[0036] Specifically, random sampling involves randomly selecting second data from all records in the first dataset. This is suitable for scenarios where the data is evenly distributed and no special conditions are required. By combining this with a hash function, it can be ensured that the second data extracted from the first dataset across multiple databases generates the same random sequence, thereby improving sampling consistency.

[0037] Stratified sampling: First data is stratified based on a certain field, and second data is randomly selected from each stratum. This method ensures that each first data point is sufficiently representative, and is particularly suitable for situations with uneven data distribution. In the embodiments of this application, sampling can be performed automatically based on the stratification logic of the field.

[0038] System sampling: This method extracts second data points from a first set of data at fixed intervals. It is suitable for situations with large datasets and the need for rapid data acquisition. The sampling interval can be automatically calculated based on the size of the first set of data and sampling requirements, and the data can be sorted according to the primary key field.

[0039] Condition-based sampling: Based on specific business needs, second data that needs to be compared is selected from the first data through WHERE conditions, and these second data are compared or analyzed in full. It is especially suitable for second data with specific filtering conditions, such as time range, numerical range, etc.

[0040] In this embodiment, to improve sampling efficiency, multi-threaded parallel processing technology can be used to execute different sampling strategies in parallel. During the execution of the sampling strategy, the first data can be automatically divided into multiple batches, and the sampling strategy can be executed simultaneously in multiple threads, greatly improving the data sampling speed. By reasonably allocating computing resources, the system can maintain high performance when processing large-scale data, avoiding the performance bottleneck in the traditional single-threaded sampling process.

[0041] For example, the system can automatically divide the first data into several parts based on the sampling batch size in the configuration table, and simultaneously perform random sampling or stratified sampling operations in multiple processing threads. After each thread completes sampling, the results are merged into the main thread to ensure the accuracy and completeness of the final results.

[0042] Step S103: Based on the second data, determine the target sampling strategy that meets the predetermined conditions from multiple sampling strategies.

[0043] After completing the sampling task, the built-in data quality assessment module automatically performs a comprehensive analysis of the second data extracted using different sampling strategies. Based on the evaluation results of each sampling strategy and the requirements of the comparison task, the system outputs the advantages and disadvantages of each strategy and recommends the optimal target sampling strategy. For example, the system might suggest using a stratified sampling strategy to ensure the representativeness of the sampled data when the data volume is large, while recommending a systematic sampling strategy to improve processing efficiency when comparing data with a shorter time range.

[0044] In this application embodiment, determining a target sampling strategy that meets predetermined conditions from multiple sampling strategies aims to perform in-depth analysis of the sampled second data, thereby evaluating the quality of the sampling strategies and quantitatively comparing the effects of different sampling strategies. The built-in data quality assessment module can intelligently evaluate the effect of each sampling strategy. Unlike traditional data analysis methods, this application, by integrating advanced statistical methods, intelligent analysis algorithms, and efficient data processing mechanisms, can quickly extract key statistical features from a large amount of second data, determine a target sampling strategy that meets predetermined conditions from multiple sampling strategies, and automatically select the optimal target sampling strategy, further improving the accuracy of data analysis and decision support capabilities.

[0045] In one embodiment, first data is sampled using multiple sampling strategies to obtain second data corresponding to each sampling strategy. This includes: analyzing the second data corresponding to each sampling strategy using a T-test, chi-square test, or analysis of variance to determine the applicability score of each sampling strategy; and determining the sampling strategy with the highest applicability score as the target sampling strategy, with the predetermined condition including the highest applicability score.

[0046] In the embodiments of this application, statistical methods, such as T-test, chi-square test or analysis of variance, can be used to quantitatively analyze the quality of the second data obtained by each sampling strategy, output the applicability score of each sampling strategy, and determine the sampling strategy with the highest applicability score as the best target sampling strategy, so as to ensure that the selected target sampling strategy is statistically superior to other sampling strategies.

[0047] In one embodiment, the first data is sampled using multiple sampling strategies to obtain the second data corresponding to each sampling strategy, including: determining the representativeness of the second data based on the central tendency, dispersion, symmetry, and kurtosis of the second data; and determining the sampling strategy corresponding to the second data with the highest representativeness as the target sampling strategy, wherein the predetermined condition includes the highest representativeness.

[0048] In this embodiment, the central tendency, dispersion, symmetry, and kurtosis of the second data sampled under each sampling strategy can be calculated to evaluate the representativeness of the sample. The sampling strategy corresponding to the second data with the highest representativeness can be determined as the target sampling strategy.

[0049] In one embodiment, before determining the representativeness of the second data based on its central tendency, dispersion, symmetry, and kurtosis, the method further includes: obtaining the mean, standard deviation, skewness, and kurtosis of the second data; determining the central tendency of the second data based on the mean; determining the dispersion of the second data based on the standard deviation; determining the symmetry of the second data based on the skewness; and determining the kurtosis of the second data based on the kurtosis.

[0050] In this embodiment, the statistical characteristics of the second data sampled by each sampling strategy, such as mean, standard deviation, skewness, and kurtosis, can be automatically calculated and compared with the overall dataset to evaluate the representativeness of the sample. A comprehensive analysis of the sampled second data is performed, including statistical indicators such as mean, standard deviation, skewness, and kurtosis. These statistics help the system understand important information such as the distribution characteristics of the sample data, outliers, and central tendency. Especially in large datasets and complex data scenarios, comprehensive analysis of multiple statistics helps to more accurately assess the representativeness of the sampling and the overall characteristics of the data.

[0051] Specifically, the mean and standard deviation of the second data obtained from each sampling strategy can be calculated to assess the central tendency and dispersion of the second data. The skewness of the second data can be determined, and the kurtosis can be determined, allowing for analysis of whether the second data conforms to the expected distribution type.

[0052] In this embodiment, the representativeness of the second data can be determined based on its central tendency, dispersion, symmetry, and kurtosis. Generally, the more concentrated the central tendency, the smaller the dispersion, the greater the symmetry, and the smaller the kurtosis, the higher the representativeness of the second data. The sampling strategy corresponding to the second data with the highest representativeness can be determined as the target sampling strategy.

[0053] In this embodiment, by employing multiple sampling strategies, intelligent data quality assessment, and efficient parallel sampling techniques, the efficiency of the sampling process is greatly improved while ensuring data representativeness. This embodiment can automatically select the optimal target sampling strategy, significantly enhancing the accuracy and stability of subsequent data comparison and analysis. Furthermore, through parallel processing and resource optimization, it can efficiently process large-scale data, meeting the needs of complex data analysis tasks and demonstrating broad application prospects.

[0054] In this embodiment, a data quality assessment module based on machine learning algorithms is also introduced. This module automatically identifies the effectiveness of different sampling strategies by training a model and provides optimization suggestions based on historical data feedback. By continuously learning from historical sampling results and successful data comparison cases, the system can automatically adjust and optimize sampling strategies to adapt to constantly changing data distributions and comparison requirements.

[0055] In this embodiment, the predictive capabilities of machine learning models can automatically identify which sampling strategies are most effective in specific scenarios and provide personalized recommendations to users. For example, the system may learn that stratified sampling achieves higher accuracy than random sampling in certain situations, and thus automatically recommend the use of stratified sampling.

[0056] This application provides an automated data quality control mechanism that evaluates the impact of various sampling strategies on data quality by comparing their performance in practical applications. The system automatically detects anomalies such as missing values, duplicate data, and format errors to identify potential data quality problems during the sampling process and provides timely improvement suggestions.

[0057] For example, during comparative analysis, if there are abnormal fluctuations or high standard deviations in the data of a certain sampling strategy, these problems will be automatically detected, and the user will be reminded that the sampling method may not be suitable for the current comparison task, and it is recommended to use other methods.

[0058] In this embodiment, advanced statistical analysis methods and a data quality assessment module enable in-depth analysis of the sampling effectiveness of each sampling strategy, comprehensively evaluating the quality of the second data extracted by each strategy, and automatically optimizing the data comparison strategy. Utilizing machine learning models, the system can self-optimize based on historical data, improving the accuracy and efficiency of the analysis process.

[0059] Step S104: Compare the second data corresponding to the target sampling strategy.

[0060] In this embodiment, the second data corresponding to the determined target sampling strategy can be compared to obtain a data comparison structure, thus completing the data comparison task. In this embodiment, parallel computing and distributed processing technologies can be used to compare the second data corresponding to the target sampling strategy, ensuring efficient completion of the analysis process even with large-scale data. By distributing the data comparison task to multiple computing nodes or threads for parallel processing, the data comparison time can be significantly shortened.

[0061] In this embodiment, data comparison tasks can be automatically divided and data comparison results can be aggregated through parallel computing nodes, ensuring computational efficiency and accuracy during the data comparison process. After each node compares a portion of the second data, the results can be aggregated to the master node, ultimately yielding a complete data comparison result. Through efficient parallel computing technology, this application can provide fast and accurate data comparison results regardless of whether facing complex multidimensional data comparisons or large-scale datasets, providing strong support for data-driven decision-making.

[0062] In one embodiment, exporting the data comparison results is the final step in the data comparison process, aiming to export the results obtained from sampling and data comparison analysis in an intuitive, easy-to-analyze, and shareable format. Unlike traditional data export methods, this embodiment designs an efficient, automated, and flexible result export mechanism that not only saves the data comparison results in commonly used formats (such as Excel) but also generates rich visualization reports according to user needs for subsequent data-driven decision-making and business operations. This step ensures efficient storage, convenient viewing, and the possibility of further analysis of the data comparison results.

[0063] In this embodiment, a data comparison result file can be automatically generated: This embodiment provides a data comparison result export module that can automatically generate corresponding files based on the results of data comparison and sampling analysis. It can automatically organize the data comparison results, generate formatted reports, and ensure that the exported file contains all important statistical indicators, evaluation results, optimization suggestions, etc., facilitating subsequent viewing and analysis by users.

[0064] The generation process of data comparison result files is fully automated, allowing users to obtain complete reports without manual intervention. File naming rules, data field formats, and report templates during the export process can all be flexibly configured according to business needs. Especially for batch data requiring large-scale processing, this application employs a batch file generation method to avoid memory pressure and performance bottlenecks that occur during single-generation.

[0065] In this embodiment, the main contents of the data comparison result file include: a detailed report of the data comparison, including difference analysis, error range, and similarity calculation for each comparison item (the second data obtained by the target sampling strategy), and may also include various statistical analysis results, such as mean, standard deviation, skewness, kurtosis, etc., as well as effectiveness evaluation and optimization suggestions for each sampling strategy. It may also include visualization charts, such as bar charts and pie charts, to help users understand the data more intuitively.

[0066] In this embodiment, rich visualization reports and data displays are provided.

[0067] Data Visualization and Chart Generation: During the export of data comparison results, this community utilizes a built-in visualization generation module to automatically convert the results into chart formats (such as bar charts, pie charts, line charts, etc.), helping users more intuitively understand the relationships, trends, and effects of different sampling strategies. Visualized reports not only improve the readability of data comparison results but also enhance the impact of data presentation.

[0068] In this embodiment, the most suitable chart type can be automatically selected based on the analysis results and business needs. Charts are then generated using a tool library (such as JFreeChart or Apache POI) and embedded into the data comparison result file. Finally, the charts and data reports are exported together. Users can directly view the charts or import them into other systems for further analysis.

[0069] In this embodiment, efficient batch result export and storage are provided.

[0070] Batch Result Export and Storage: This application employs a batch result export mechanism. When processing large-scale data, it can export data comparison results in batches to ensure that the system's response time is not affected or memory overflow issues occur due to the excessive data volume of a single export task. Batch exported files are automatically named according to date, task number, or other business requirements, ensuring orderly storage and easy management. This is particularly suitable for industries that require processing large amounts of data (such as finance, healthcare, and retail), automatically exporting and storing results to a specified directory according to preset rules. It supports multiple output formats, such as Excel, CSV, and JSON, to meet the needs of different application scenarios.

[0071] In this embodiment of the application, highly configurable export options are provided.

[0072] Highly configurable export options: The results export module of this application supports highly configurable export options, allowing users to flexibly choose the export content, report format, and chart type according to their needs. Users can set parameters such as specific fields, report templates, and file save paths to ensure personalized and flexible results export. For example, users can choose to export only the analysis results of specific fields, or choose to export the complete statistical analysis report and charts. Furthermore, this application supports an automated file saving mechanism, which can save the generated report to a specified cloud storage service or local server, ensuring file security and convenient access.

[0073] This application's automated result generation, data visualization reports, batch data export, and flexible configuration options enable the efficient output of complete analysis reports. Furthermore, intuitive charts and detailed data allow for further analysis of data comparison results. This result export mechanism not only significantly improves the convenience of data analysis but also maintains the system's efficiency and stability in large-scale data processing scenarios. By automating the generation of high-quality reports, this application greatly enhances the efficiency of data analysis and decision support, and has broad application prospects.

[0074] The data comparison method of this application will be explained in detail below with specific examples.

[0075] Example 1: Verification of data migration for financial institutions.

[0076] This example illustrates a commercial bank's need to migrate core transaction data from an existing GaussDB database to a newly built PostgreSQL database cluster during a system upgrade. Data consistency verification is achieved through the following steps: 1. Database configuration phase: Establish dual database connections through the Spring Boot framework, configure GaussDB connection parameters (including SSL encryption verification), and configure PostgreSQL high-availability cluster connections.

[0077] 2. Configuration table loading: Reads the comparison rules of the transaction log (TRANS_LOG) from the DB_CONFIG table in the system management database, including: ① List of key fields: Transaction serial number, account ID, transaction amount, status code.

[0078] ②Sampling criteria: Key transactions with a transaction amount exceeding RMB 100,000, within the transaction time range of **** year ** month to ** month.

[0079] ③ Stratified sampling setup: Divide the data into layers according to account type (corporate / personal), and extract 5% of the data from each layer.

[0080] 3. Sampling and Comparison Execution: A stratified sampling method was used to sample and compare 58 million transaction records before and after the migration. The program completed this automatically. ① Field-level consistency verification (amount precision, status code mapping).

[0081] ② Data integrity verification (missing record detection).

[0082] ③ Transaction consistency check (atomicity of transactions involving multiple tables).

[0083] 4. Analysis Output: A sampling evaluation report is generated, showing that the chi-square test p-value for stratified sampling reaches 0.92, recommending this sampling strategy. Finally, the detailed anomaly records are exported to Excel, containing 342 records of monetary deviations and 89 anomalies with status codes.

[0084] Example 2: Cross-system integration of medical data.

[0085] This example demonstrates how a regional healthcare consortium can integrate patient data from three hospitals (using PostgreSQL, MySQL, and Oracle respectively) to build a unified data center.

[0086] 1. Multi-database configuration: Dynamically loads database connection configurations for each hospital, establishing a heterogeneous database connection pool. JDBC batch processing optimization parameters are specifically configured for Oracle databases.

[0087] 2. Intelligent Sampling Strategy: Based on the characteristics of the medical record form (MEDICAL_RECORDS): ① A full comparison was used for critically ill patients (ICD-10 code J96.x).

[0088] ② For ordinary cases, systematic sampling based on diagnosis time (interval of 100 cases) was used.

[0089] ③ The test report form is obtained by random sampling (3%).

[0090] 3. Difference Analysis: Automatically detects: ①Differences in drug coding mapping (12 drugs have multiple coding issues).

[0091] ② The time formats are inconsistent (two hospitals use the UTC time zone, and one uses the local time zone).

[0092] ③ Differences in precision of numerical fields (different rules for retaining decimal places in test results).

[0093] 4. Output Optimization Suggestions: Statistics show that full comparison takes 78% longer than stratified sampling, but the detection rate only improves by 2.3%. It is recommended to use a hybrid sampling strategy for key tables and finally generate an Excel template containing data mapping rules for ETL use.

[0094] Example 3: Global synchronous verification of logistics orders.

[0095] This example illustrates how a multinational logistics company needs to ensure real-time synchronization of order data between Zone 1 (GaussDB) and Zone 2 (PostgreSQL).

[0096] 1. Dynamic Configuration: The system obtains the latest database connection information in real time through the configuration center, supporting an automatic reconnection mechanism during master-slave failover. Configure the comparison rules for the order table (LOGISTICS_ORDERS): ①Key fields: Waybill number, logistics status, timestamp.

[0097] ② Incremental comparison window: Data from the most recent 24 hours.

[0098] ③ Sampling frequency: Automatically executed once per hour.

[0099] 2. Efficient sampling mechanism: Employing a time window-based systematic sampling method. ① Samples are drawn in batches at 10-minute intervals.

[0100] ② Perform distributed sampling by combining the hash value of the waybill number.

[0101] ③ Automatically trigger a full comparison for orders in abnormal status (such as "delayed").

[0102] 3. Real-time analysis: The program has a built-in streaming computing module for statistical display. ① The standard deviation of synchronization delay was optimized from ±15s initially to ±2.3s.

[0103] ② The inconsistency rate remained stable below 0.003%.

[0104] ③ The sampling efficiency is 40 times higher than that of the full comparison.

[0105] 4. Visualized output: Automatically generates Excel reports containing time-series trend charts, marking peak periods of synchronization anomalies (daily UTC 03:00-05:00), providing data support for operation and maintenance optimization.

[0106] It should be noted that the data comparison method provided in this application embodiment can be executed by a data comparison device or a control module within that data comparison device for executing the data comparison method. This application embodiment uses the execution of the data comparison method by a data comparison device as an example to illustrate the data comparison device provided in this application embodiment.

[0107] Figure 2 This is a schematic diagram of the data comparison device according to an embodiment of this application. Figure 2 As shown, the data comparison device 200 includes: an acquisition module 210, a sampling module 220, a determination module 230, and a comparison module 240.

[0108] The acquisition module 210 is used to acquire first data to be compared from multiple databases; the sampling module 220 is used to sample the first data using multiple sampling strategies to obtain second data corresponding to each sampling strategy; the determination module 230 is used to determine the target sampling strategy that meets the predetermined conditions from multiple sampling strategies based on the second data; and the comparison module 240 is used to compare the second data corresponding to the target sampling strategy.

[0109] In one embodiment, the determining module 230 is used to analyze the second data corresponding to each sampling strategy through T-test, chi-square test or analysis of variance to determine the applicability score of each sampling strategy; and to determine the sampling strategy with the highest applicability score as the target sampling strategy, wherein the predetermined condition includes the highest applicability score.

[0110] In one embodiment, the determining module 230 is used to determine the representativeness of the second data based on the central tendency, dispersion, symmetry and kurtosis of the second data; and to determine the sampling strategy corresponding to the second data with the highest representativeness as the target sampling strategy, wherein the predetermined condition includes the highest representativeness.

[0111] In one embodiment, the determining module 230 is further configured to acquire the mean, standard deviation, skewness, and kurtosis of the second data; determine the central tendency of the second data based on the mean; determine the data dispersion of the second data based on the standard deviation; determine the symmetry of the second data based on the skewness; and determine the kurtosis of the second data based on the kurtosis.

[0112] In one embodiment, the plurality of sampling strategies includes at least: random sampling, stratified sampling, systematic sampling, and condition-based sampling.

[0113] In one embodiment, the determining module 230 is used to obtain pre-created target connections to multiple databases from the connection pool; and connect to multiple databases through the target connections.

[0114] In one embodiment, the determining module 230 is used to obtain first data from multiple databases according to a configuration table, wherein the configuration table includes database identifiers of multiple databases, a data table corresponding to the first data, key fields of the first data, and filtering conditions for the first data.

[0115] The data comparison device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0116] The data comparison device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0117] The data comparison device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0118] Optionally, such as Figure 3 As shown in the illustration, this application embodiment also provides an electronic device 300, including a processor 301 and a memory 302. The memory 302 stores a program or instructions that can run on the processor 301. When the program or instructions are executed by the processor 301, they perform the following: obtaining first data to be compared from multiple databases; sampling the first data using multiple sampling strategies to obtain second data corresponding to each sampling strategy; determining a target sampling strategy that meets predetermined conditions from the multiple sampling strategies based on the second data; and comparing the second data corresponding to the target sampling strategy.

[0119] In one embodiment, the second data corresponding to each sampling strategy is analyzed by T-test, chi-square test or analysis of variance to determine the applicability score of each sampling strategy; the sampling strategy with the highest applicability score is determined as the target sampling strategy, and the predetermined condition includes the highest applicability score.

[0120] In one embodiment, the representativeness of the second data is determined based on the central tendency, dispersion, symmetry, and kurtosis of the second data; the sampling strategy corresponding to the second data with the highest representativeness is determined as the target sampling strategy, and the predetermined condition includes the highest representativeness.

[0121] In one embodiment, before determining the representativeness of the second data based on its central tendency, dispersion, symmetry, and kurtosis, the mean, standard deviation, skewness, and kurtosis of the second data are obtained; the central tendency of the second data is determined based on the mean; the dispersion of the second data is determined based on the standard deviation; the symmetry of the second data is determined based on the skewness; and the kurtosis of the second data is determined based on the kurtosis.

[0122] In one embodiment, the plurality of sampling strategies includes at least: random sampling, stratified sampling, systematic sampling, and condition-based sampling.

[0123] In one embodiment, before retrieving the first data to be compared from multiple databases, a pre-created target connection to multiple databases is obtained from a connection pool; the multiple databases are then connected via the target connection.

[0124] In one embodiment, first data is obtained from multiple databases according to a configuration table, the configuration table including database identifiers of multiple databases, data tables corresponding to the first data, key fields of the first data, and filtering conditions for the first data.

[0125] The specific execution steps can be found in the various steps of the above data comparison method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0126] It should be noted that the electronic devices in the embodiments of this application include: servers, terminals, or other devices besides terminals.

[0127] The above electronic device structure does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or arrange them differently. For example, an input unit may include a Graphics Processing Unit (GPU) and a microphone, and a display unit may use a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar display panels. User input units include at least one of a touch panel and other input devices. A touch panel is also called a touchscreen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be elaborated further here.

[0128] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0129] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0130] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data comparison method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0131] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media such as ROM, RAM, magnetic disk, or optical disk.

[0132] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0134] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data comparison method, characterized in that, include: Retrieve the first data to be compared from multiple databases; The first data is sampled using multiple sampling strategies to obtain second data corresponding to each sampling strategy. Based on the second data, a target sampling strategy that meets the predetermined conditions is determined from a variety of sampling strategies; The second data corresponding to the target sampling strategy is compared.

2. The method according to claim 1, characterized in that, The step of sampling the first data using multiple sampling strategies to obtain second data corresponding to each sampling strategy includes: The second data corresponding to each of the sampling strategies is analyzed by t-test, chi-square test or analysis of variance to determine the fitness score of each of the sampling strategies. The sampling strategy with the highest fitness score is determined as the target sampling strategy, and the predetermined condition includes the highest fitness score.

3. The method according to claim 1, characterized in that, The step of sampling the first data using multiple sampling strategies to obtain second data corresponding to each sampling strategy includes: The representativeness of the second data is determined based on its central tendency, dispersion, symmetry, and kurtosis. The sampling strategy corresponding to the second data with the highest representativeness is determined as the target sampling strategy, and the predetermined condition includes the highest representativeness.

4. The method according to claim 3, characterized in that, Before determining the representativeness of the second data based on its central tendency, dispersion, symmetry, and kurtosis, the method further includes: Obtain the mean, standard deviation, skewness, and kurtosis of the second data; Based on the mean, determine the central tendency of the second data; The degree of data dispersion of the second data is determined based on the standard deviation. The symmetry of the second data is determined based on the skewness. The kurtosis of the second data is determined based on the kurtosis.

5. The method according to claim 1, characterized in that, The various sampling strategies include at least: Random sampling, stratified sampling, systematic sampling, and condition-based sampling.

6. The method according to claim 1, characterized in that, Before obtaining the first data to be compared from multiple databases, the method further includes: Obtain pre-created target connections to the multiple databases from the connection pool; The target connection connects to multiple databases.

7. The method according to claim 1, characterized in that, The step of obtaining the first data to be compared from multiple databases includes: According to the configuration table, the first data is obtained from multiple databases. The configuration table includes the database identifiers of the multiple databases, the data table corresponding to the first data, the key fields of the first data, and the filtering conditions of the first data.

8. A data comparison device, characterized in that, include: The acquisition module is used to retrieve the first data to be compared from multiple databases; The sampling module is used to sample the first data using multiple sampling strategies to obtain second data corresponding to each sampling strategy. The determination module is used to determine a target sampling strategy that meets predetermined conditions from multiple sampling strategies based on the second data. The comparison module is used to compare the second data corresponding to the target sampling strategy.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the data comparison method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data comparison method as described in any one of claims 1-7.

Images