Heterogeneous database data snapshot comparison method, device, equipment, medium and product

Abstract

The invention discloses a heterogeneous database data snapshot comparison method, device and equipment, a medium and a product. The method comprises the following steps: generating a cross-database mapping rule between a source end database and a target end database by adopting a language model; constructing a distributed comparison task pool based on Kubernetes, and carrying out comparison task parallel scheduling according to data blocks; and executing a comparison task, and reading corresponding partition data from the source end database and the target end database based on the cross-database mapping rule for comparison. According to the technical scheme provided by the invention, the cross-database Schema mapping is automatically converted through the large model, so that the heterogeneous database comparison efficiency is greatly improved, and meanwhile, the labor cost is also reduced. In addition, by utilizing elastic resource scheduling of the cloud native architecture, on-demand utilization of resources is realized, and elastic expansion of burst tasks can be supported.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of database technology, and in particular to a method, apparatus, device, medium and product for comparing data snapshots of heterogeneous databases. Background Technology

[0002] For cross-architecture data comparison of multiple types of databases, traditional tools rely on manual configuration of mapping rules. At the same time, existing NLP models have not been specifically optimized for database schema semantics, resulting in low automation of cross-database schema mapping. This leads to low efficiency in heterogeneous database comparison and high manual costs. Summary of the Invention

[0003] This invention provides a method, apparatus, device, medium, and product for comparing data snapshots of heterogeneous databases, in order to improve the efficiency of heterogeneous database comparison and reduce labor costs.

[0004] In a first aspect, embodiments of the present invention provide a method for comparing data snapshots of heterogeneous databases, the method comprising:

[0005] A language model is used to generate cross-database mapping rules between the source and target databases;

[0006] A distributed comparison task pool is built based on Kubernetes, and comparison tasks are scheduled in parallel according to data blocks.

[0007] The comparison task is executed by reading the corresponding partition data from the source database and the target database based on the cross-database mapping rules for comparison.

[0008] Optionally, the method further includes:

[0009] A time-series model is used to predict the probability of future changes to each data block based on historical change logs;

[0010] The data is dynamically divided into blocks based on the change probability and the original block size.

[0011] Optionally, the parallel scheduling of the comparison task based on data blocks includes:

[0012] Data blocks are scheduled according to preset priorities. Data blocks undergoing schema changes have high priority, data blocks with a change probability greater than a preset threshold have medium priority, and other data blocks have low priority.

[0013] Optionally, the method further includes:

[0014] Real-time collection of resource usage metrics for each distributed node;

[0015] The parallelism of scheduled tasks is dynamically adjusted based on the resource consumption index.

[0016] Optionally, the scheduling process supports a hybrid scheduling of GPU-accelerated and CPU-intensive tasks.

[0017] Optionally, the method further includes:

[0018] A reinforcement learning model is used to optimize the data comparison strategy. The objective function is to minimize the comparison time and false alarm rate. The state space includes the block change probability, transaction latency and network bandwidth. The action space includes adjusting the comparison time window and merging adjacent blocks. The reward function is a weighted value of the comparison accuracy and resource consumption.

[0019] Secondly, embodiments of the present invention also provide a heterogeneous database data snapshot comparison device, the device comprising:

[0020] The mapping rule generation module is used to generate cross-database mapping rules between the source database and the target database using a language model.

[0021] The comparison task scheduling module is used to build a distributed comparison task pool based on Kubernetes and to schedule comparison tasks in parallel according to data blocks.

[0022] The comparison task execution module is used to execute comparison tasks, and reads the corresponding partition data from the source database and the target database for comparison based on the cross-database mapping rules.

[0023] Thirdly, embodiments of the present invention also provide a computer device, the computer device comprising:

[0024] One or more processors;

[0025] Memory, used to store one or more programs;

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the heterogeneous database data snapshot comparison method provided in any embodiment of the present invention.

[0027] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the heterogeneous database data snapshot comparison method provided in any embodiment of the present invention.

[0028] Fifthly, embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the heterogeneous database data snapshot comparison method provided in any embodiment of the present invention.

[0029] This invention provides a method for comparing data snapshots of heterogeneous databases. First, a language model is used to generate cross-database mapping rules between the source and target databases. Then, a distributed comparison task pool is built based on Kubernetes, and comparison tasks are scheduled in parallel according to data blocks. Next, the comparison tasks are executed, reading corresponding partition data from the source and target databases for comparison based on the generated cross-database mapping rules. The heterogeneous database snapshot comparison method provided by this invention significantly improves the efficiency of heterogeneous database comparison by automatically converting cross-database schema mappings using a large model, while also reducing manual costs. Furthermore, by utilizing the elastic resource scheduling of cloud-native architecture, resources are utilized on demand and can support elastic scaling for sudden tasks. Attached Figure Description

[0030] Figure 1 This is a flowchart of the heterogeneous database data snapshot comparison method provided in Embodiment 1 of the present invention;

[0031] Figure 2 This is a schematic diagram of the heterogeneous database data snapshot comparison device provided in Embodiment 2 of the present invention;

[0032] Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0033] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0034] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0035] Example 1

[0036] Figure 1This is a flowchart of a heterogeneous database snapshot comparison method provided in Embodiment 1 of the present invention. This embodiment is applicable to snapshot comparison between heterogeneous databases, and can also be used for snapshot comparison between homogeneous databases. Specifically, it can be used to implement low-resource, high-performance data verification during database migration, such as cross-cloud database migration and multi-architecture data consistency verification. It meets the needs of enterprise hybrid cloud environments and can be packaged as a DbaaS service (such as enhanced features of AWS DMS). This method can be executed by the heterogeneous database snapshot comparison device provided in this embodiment of the invention. This device can be implemented in hardware and / or software, and is generally integrated into a computer device. Figure 1 As shown, the specific steps include the following:

[0037] S11. Use a language model to generate cross-database mapping rules between the source database and the target database.

[0038] S12. Build a distributed comparison task pool based on Kubernetes, and schedule comparison tasks in parallel according to data blocks.

[0039] S13. Execute the comparison task, and read the corresponding partition data from the source database and the target database based on the cross-database mapping rules for comparison.

[0040] Specifically, firstly, cross-database mapping rules can be constructed between the source and target databases to be compared, enabling automatic schema mapping. Both the source and target databases can be any database, such as MySQL, Oracle, TiDB, PostgreSQL, MongoDB, Redis, and CockroachDB. Generative AI, such as CodeT5, Codex, and LLaMA, based on Transformer code generation models, can be used to analyze the differences in schemas between heterogeneous databases by parsing DDL statements and document structures, thereby automatically generating lightweight cross-database mapping rules. Based on these rules, mutual conversion between the source and target database schemas can be achieved; for example, a nested document structure from MongoDB can be converted into a multi-table join schema in MySQL.

[0041] For the language model used, a large number (e.g., 500,000) of cross-database schema transformation pairs (e.g., MySQL↔ TiDB, Oracle↔ CockroachDB, etc.) can be pre-collected as a training dataset. Data augmentation can be performed by randomly inserting annotations, changing the field order, and type conversion interference, and then the language model can be trained. The example model input (MySQL Schema) is CREATE TABLE users(id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(255) NOT NULL), and the example model output (TiDB Schema) is CREATE TABLE users(id BIGINT AUTO_RANDOM PRIMARY KEY, name VARCHAR(255) NOT NULL). For schema collection of relational databases, taking MySQL as an example, the MySQL DDL statement can be obtained by SHOW CREATE TABLE, and constraint information can be extracted by parsing ALL_CONSTRAINTS and ALL_CONS_COLUMNS. For schema collection in distributed databases, taking TiDB as an example, you can obtain region distribution metadata by calling the TiDB PD API and retrieve DDL change logs by querying INFORMATION_SCHEMA.TIDB_DDL_HISTORY.

[0042] Then, the trained language model can be used to perform semantic parsing of the schemas of the source and target databases, generating a unified intermediate schema representation (ISR). For example, the input includes the source database schema (such as MySQL's DDL statements), the target database schema (such as TiDB's DDLHistory logs), and historical schema change logs. The output is a cross-database field-level mapping relationship, such as mapping MySQL's VARCHAR(255) to MongoDB's string type. The language model can be fine-tuned using open-source frameworks (such as HuggingFace and TensorFlow) to support database-specific syntax, such as the precision difference between Oracle's NUMBER and PostgreSQL's DECIMAL. Furthermore, statistical learning algorithms (such as PCA and TSNE) can be used to analyze the field distribution, index patterns, and data skew of the table structure, generating data feature vectors for subsequent block-based strategy optimization. For example, the feature dimensions include the field distribution as SELECT COUNT(*) / AVG(length(name)) FROM users, the index mode as SHOWINDEX FROM orders, and the data skew as SELECT COUNT(*) GROUP BY user_id ORDER BY 1DESC LIMIT 10. Then, vector encoding can be performed, PCA can be used to reduce the dimensionality to 128 dimensions, and the data can be stored in the Redis Cluster cache for later use.

[0043] After determining the cross-database mapping rules, a distributed comparison task pool can be built based on Kubernetes, and containerized comparison units (Comparator Pods) can be scheduled in parallel to execute comparison tasks according to data blocks. Container orchestration systems such as Kubernetes support dynamic scaling, and can automatically allocate comparison task instances according to data volume and load. At the same time, it can be combined with a serverless architecture (such as AWS Lambda) to achieve on-demand computing, allocating resources only when comparison tasks are triggered, thereby reducing idle costs. Specifically, serverless comparison units (Comparator Functions) can be deployed in the form of lightweight containers (Docker), started on demand and automatically destroyed. For example, block comparison tasks can be split into stateless functions, reducing the execution time of a single execution from minutes to seconds. Optionally, the scheduling process supports the hybrid scheduling of GPU-accelerated (e.g., for hash calculation) and CPU-intensive tasks (e.g., for text comparison), and combined with the use of a cloud-native elastic comparison engine, resource utilization can be significantly improved. The comparison task can then be dynamically allocated to different nodes for execution via a task dispatcher. Specifically, based on cross-database mapping rules, corresponding partition data is read from the source and target databases for comparison to obtain the comparison results, which can return difference markers. The comparison process can employ a hybrid consistency verification algorithm. For static data, a Merkle Tree is used to quickly locate differences, specifically by using Guava's Hashing library to reduce construction time. For dynamic data, field-level comparison can be performed based on a similarity threshold (e.g., 0.9). Furthermore, the comparison granularity can be adjusted according to the change probability of the corresponding block, such as from row-level to field-level changes. Additionally, TiDB's MVCC multi-version verification is supported, ensuring that data from the same point in time is read by reading snapshot data with a specified TSO timestamp, thus guaranteeing transactional consistency in cross-database comparisons. For further comparison tasks between relational and distributed databases, time-series alignment algorithms can be used to synchronize the Binlog position of the relational database with the CDC timestamp of the distributed database. This eliminates misjudgments caused by transaction commit delays. For example, the alignment strategy involves waiting for the TiDB TSO to exceed the MySQL Binlog time + 500ms, and compensating for network latency, such as measuring network round-trip latency and compensating for half-trip latency. Additionally, a distributed cache (Redis Cluster) can be introduced to store frequently accessed schema mapping rules and intermediate hash values ​​to reduce cross-database query overhead.

[0044] In an optional implementation, the method further includes: real-time collection of resource usage metrics for each distributed node; and dynamic adjustment of the task parallelism based on the resource usage metrics. Specifically, a resource monitor can be used to collect real-time resource usage metrics such as CPU, memory, and network latency for each node to dynamically adjust the task parallelism. This allows for dynamic adaptation to load fluctuations and addresses the problem that traditional schedulers (such as the default Kubernetes scheduler) lack the ability to perceive database-specific load patterns when the computational density and network I / O requirements of comparison tasks differ significantly. For example, CPU utilization can be monitored using HPA (Horizontal Pod Autoscaler) to increase the parallelism of the comparison task from 10 to 100 Pods.

[0045] Based on the above technical solution, optionally, the method further includes: using a time series model to predict the probability of future changes to each data block based on historical change logs; and dynamically dividing the data blocks according to the change probability and the original block size.

[0046] Specifically, time-series models such as LSTM, Transformer, and DeepAR can be used to analyze historical data change patterns, dynamically identify high-frequency change regions, and dynamically divide the data into blocks. High-frequency change regions can also be prioritized for comparison; for example, in e-commerce promotional scenarios, predicting change hotspots in the "order status" field allows for priority comparison of related blocks. Compared to traditional block-sharing methods (such as fixed-size blocks or range sharding), this approach better adapts to the dynamic changes in data hotspots and significantly reduces comparison time. Specifically, the probability of change for each original data block in a preset future time period (such as the next hour) can be predicted based on historical change logs (such as Binlog and Oplog) from the most recent preset historical period (such as the past 24 hours), such as the QPS trend of TiDB hotspot regions. Then, dynamic block-sharing is performed by combining the block size and change probability of the original data blocks. The block-sharing process can be performed according to a specified time window, a specified block size, or a specified block base; for example, a region with a change probability greater than 80% can be split into smaller 1MB blocks.

[0047] For example, the `tidb_region_status` metric can be collected using Prometheus as training data for a time-series model. Real-time data such as the region write volume (in MB / s) over the past 24 hours (e.g., 0.1, 0.2, 0.5, ..., 1.2) can be used as model input to predict the probability of change in the next hour (e.g., "region-1023": 0.85, "region-456": 0.32). The model can then be continuously optimized, such as by adjusting hyperparameters like the number of LSTM layers and the learning rate using Bayesian optimization algorithms. After obtaining the change probability of each data block, high-frequency data blocks with change probabilities exceeding a preset threshold (e.g., 0.7) can be segmented according to a preset time window (e.g., 15 minutes). Then, large data blocks exceeding a preset size (e.g., 100MB) in the remaining data blocks can be segmented according to a preset cardinality (e.g., 10). The remaining data blocks can remain intact. After segmentation, the segment metadata can be stored in etcd.

[0048] Based on the above technical solution, optionally, the parallel scheduling of comparison tasks by data blocks includes: scheduling according to preset priorities, with data blocks in schema changes having high priority, data blocks with a change probability greater than a preset threshold having medium priority, and other data blocks having low priority.

[0049] Specifically, a topology-aware scheduling strategy can be used to extend the configuration of the Kubernetes scheduler to achieve priority scheduling, thereby further reducing cross-database query latency. For priority queue management, data blocks involved in schema changes can be divided into high-priority segments and compared in real time. Data blocks with a change probability greater than a preset threshold (e.g., 0.7) can be divided into medium-priority segments and compared within 15 minutes. Other data blocks (cold data) can be batch-processed as low-priority segments. Furthermore, transaction latency can be considered, with replicas with higher transaction latency being processed as higher priority, specifically higher than cold data but lower than hot data.

[0050] Based on the above technical solution, optionally, the method further includes: optimizing the data comparison strategy using a reinforcement learning model, with the objective function being to minimize the comparison time and false alarm rate; the state space including block change probability, transaction latency, and network bandwidth; the action space including adjusting the comparison time window and merging adjacent blocks; and the reward function being a weighted value of comparison accuracy and resource consumption. Specifically, the Q-Learning algorithm can be used to dynamically adjust the comparison path to further reduce the number of cross-database queries. For example, when comparing MySQL and Redis, data with consistent primary key indexes can be compared first, reducing the overhead of full table scans. An example of adjusting the comparison time window is to delay reading the TiDB replica by 300ms.

[0051] Building upon the aforementioned technical solutions, the schema change status of the distributed database can also be monitored, such as the progress of TiDB's ADD INDEX background task, to dynamically adjust the comparison rules. Specifically, if the target database schema change is not yet complete (e.g., index building is underway), the comparison of relevant fields is skipped; if the schema versions of the source and target databases are inconsistent, the comparison is automatically rolled back to the common version.

[0052] Building upon the aforementioned technical solutions, large language models (such as LLM) can be utilized for discrepancy attribution and repair suggestions to further reduce manual investigation time and improve repair accuracy. Specifically, abnormal patterns in the comparison results, such as data skew and field type conflicts, can be analyzed. Multi-dimensional comparisons can be performed, including statistical discrepancies (table level) such as `SELECT COUNT(*) FROM mysql_table MINUS SELECT COUNT(*) FROM tidb_table`, and checksum discrepancies (field level) such as `SELECT SUM(CRC32(name)) FROM users GROUP BY gender`. Then, causal inference models, such as Do-Calculus, can be used to analyze the root causes of discrepancies (e.g., network / schema / data). For example, P(difference|network latency) = 0.82, suggesting 82% is due to synchronization latency; P(difference|schema conflict) = 0.15, suggesting 15% is due to type conversion errors. Furthermore, based on the presumed results from LlaMA-65B, a repair SQL script can be automatically generated. For example, if the problem is a difference between MySQL's {mysql_schema} and TiDB's {tidb_schema}, the requirement is to generate an executable ALTER statement to maintain data compatibility, with the corresponding output being `ALTER TABLE orders MODIFY COLUMN amount DECIMAL(20,6)`. Then, a zero-downtime repair process can be used. For example, first, a shadow table is created, and the old and new tables are written in parallel. Then, the shadow table is compared with the target database. If the difference is less than 0.1%, the table name is atomically switched; otherwise, a rollback is performed and an alert is issued.

[0053] In summary, by deeply integrating cloud-native elastic architecture with AI intelligent decision-making, comparison efficiency can be improved by orders of magnitude while ensuring data consistency, meeting the stringent requirements of financial-grade heterogeneous database operation and maintenance. Specifically, AI-predicted change probabilities reduce invalid comparison computations, improving efficiency in massive data scenarios. Large models assist in schema adaptation and provide repair suggestions, reducing manual intervention. Furthermore, dynamic block acceleration and network transmission optimization enable real-time snapshot comparisons for terabyte-scale data volumes, addressing issues such as low comparison efficiency, insufficient resource utilization, high reliance on manual intervention, and inadequate reliability and fault tolerance in existing heterogeneous databases. Moreover, federated learning can be combined to protect data privacy in multi-tenant scenarios. Vector databases (such as FAISS and Milvus) can be introduced to accelerate schema similarity matching. Vector databases support high-dimensional data feature storage and fast retrieval for metadata indexing in comparison tasks. The distributed computing framework Apache Spark can be used to accelerate hash calculations and difference analysis, combined with memory acceleration layers such as Alluxio to reduce disk I / O, and can interact with the database kernel to further reduce comparison latency using Change Data Capture (CDC).

[0054] The technical solution provided in this invention first uses a language model to generate cross-database mapping rules between the source and target databases. Then, a distributed comparison task pool is built based on Kubernetes, and comparison tasks are scheduled in parallel according to data blocks. Next, the comparison tasks are executed, reading corresponding partition data from the source and target databases for comparison based on the generated cross-database mapping rules. By automatically converting the cross-database schema mapping using a large model, the efficiency of heterogeneous database comparison is significantly improved, while also reducing manual costs. Furthermore, by utilizing the elastic resource scheduling of cloud-native architecture, resources are utilized on demand, and elastic scaling for sudden tasks is supported.

[0055] Example 2

[0056] Figure 2 This is a schematic diagram of the heterogeneous database snapshot comparison device provided in Embodiment 2 of the present invention. This device can be implemented in hardware and / or software, and is generally integrated into a computer device to execute the heterogeneous database snapshot comparison method provided in any embodiment of the present invention. Figure 2 As shown, the device includes:

[0057] Mapping rule generation module 21 is used to generate cross-database mapping rules between the source database and the target database using a language model;

[0058] The comparison task scheduling module 22 is used to build a distributed comparison task pool based on Kubernetes and to perform parallel scheduling of comparison tasks according to data blocks.

[0059] The comparison task execution module 23 is used to execute the comparison task, and read the corresponding partition data from the source database and the target database based on the cross-database mapping rules for comparison.

[0060] The technical solution provided in this invention first uses a language model to generate cross-database mapping rules between the source and target databases. Then, a distributed comparison task pool is built based on Kubernetes, and comparison tasks are scheduled in parallel according to data blocks. Next, the comparison tasks are executed, reading corresponding partition data from the source and target databases for comparison based on the generated cross-database mapping rules. By automatically converting the cross-database schema mapping using a large model, the efficiency of heterogeneous database comparison is significantly improved, while also reducing manual costs. Furthermore, by utilizing the elastic resource scheduling of cloud-native architecture, resources are utilized on demand, and elastic scaling for sudden tasks is supported.

[0061] Based on the above technical solution, optionally, the device further includes:

[0062] The change probability prediction module is used to predict the probability of future changes to each data block based on historical change logs using a time series model.

[0063] The data block partitioning module is used to dynamically partition the data blocks according to the change probability and the original block size.

[0064] Based on the above technical solution, optionally, the comparison task scheduling module 22 is specifically used for:

[0065] Data blocks are scheduled according to preset priorities. Data blocks undergoing schema changes have high priority, data blocks with a change probability greater than a preset threshold have medium priority, and other data blocks have low priority.

[0066] Based on the above technical solution, optionally, the device further includes:

[0067] The resource usage acquisition module is used to collect resource usage indicators of each distributed node in real time.

[0068] The task parallelism adjustment module is used to dynamically adjust the parallelism of scheduled tasks based on the resource consumption index.

[0069] Based on the above technical solutions, optionally, the scheduling process supports hybrid scheduling of GPU-accelerated and CPU-intensive tasks.

[0070] Based on the above technical solution, optionally, the device further includes:

[0071] The comparison strategy optimization module is used to optimize the data comparison strategy using a reinforcement learning model. The objective function is to minimize the comparison time and false alarm rate. The state space includes block change probability, transaction latency and network bandwidth. The action space includes adjusting the comparison time window and merging adjacent blocks. The reward function is a weighted value of comparison accuracy and resource consumption.

[0072] The heterogeneous database snapshot comparison device provided in this embodiment of the invention can execute the heterogeneous database snapshot comparison method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0073] It is worth noting that in the embodiments of the heterogeneous database data snapshot comparison device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0074] Example 3

[0075] Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention, showing a block diagram of an exemplary computer device suitable for implementing the embodiments of the present invention. Figure 3 The computer device shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. Figure 3 As shown, the computer device includes a processor 31, a memory 32, an input device 33, and an output device 34; the number of processors 31 in the computer device can be one or more. Figure 3 Taking a processor 31 as an example, the processor 31, memory 32, input device 33, and output device 34 in a computer device can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0076] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the heterogeneous database data snapshot comparison method in this embodiment of the invention (e.g., the mapping rule generation module 21, the comparison task scheduling module 22, and the comparison task execution module 23 in the heterogeneous database data snapshot comparison device). The processor 31 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 32, thereby implementing the aforementioned heterogeneous database data snapshot comparison method.

[0077] The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 32 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 32 may further include memory remotely located relative to the processor 31, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0078] Input device 33 can be used to acquire source and target database data for comparison, and to generate key signal inputs related to user settings and function control of the computer device. Output device 34 can be used to output comparison results, etc.

[0079] Example 4

[0080] Embodiment 4 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a heterogeneous database data snapshot comparison method, the method comprising:

[0081] A language model is used to generate cross-database mapping rules between the source and target databases;

[0082] A distributed comparison task pool is built based on Kubernetes, and comparison tasks are scheduled in parallel according to data blocks.

[0083] The comparison task is executed by reading the corresponding partition data from the source database and the target database based on the cross-database mapping rules for comparison.

[0084] Storage media can be any type of memory device or storage device. The term "storage media" is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a computer system in which the program is executed, or may reside in a different second computer system connected to the computer system via a network (such as the Internet). The second computer system can provide program instructions to the computer for execution. The term "storage media" can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) that can be executed by one or more processors.

[0085] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also perform related operations in the heterogeneous database data snapshot comparison method provided in any embodiment of the present invention.

[0086] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0087] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0088] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0089] Example 5

[0090] Embodiment 5 of the present invention also provides a computer program product, which includes a computer program (also referred to as code or instructions). The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it is used to execute the heterogeneous database data snapshot comparison method provided in any of the above embodiments, and has the corresponding beneficial effects of the execution method.

[0091] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for comparing data snapshots from heterogeneous databases, characterized in that, include: A language model is used to generate cross-database mapping rules between the source and target databases; A distributed comparison task pool is built based on Kubernetes, and comparison tasks are scheduled in parallel according to data blocks. The comparison task is executed by reading the corresponding partition data from the source database and the target database based on the cross-database mapping rules for comparison.

2. The heterogeneous database snapshot comparison method according to claim 1, characterized in that, The method further includes: A time-series model is used to predict the probability of future changes to each data block based on historical change logs; The data is dynamically divided into blocks based on the change probability and the original block size.

3. The heterogeneous database snapshot comparison method according to claim 2, characterized in that, The parallel scheduling of comparison tasks based on data blocks includes: Data blocks are scheduled according to preset priorities. Data blocks undergoing schema changes have high priority, data blocks with a change probability greater than a preset threshold have medium priority, and other data blocks have low priority.

4. The heterogeneous database snapshot comparison method according to claim 1, characterized in that, The method further includes: Real-time collection of resource usage metrics for each distributed node; The parallelism of scheduled tasks is dynamically adjusted based on the resource consumption index.

5. The heterogeneous database snapshot comparison method according to claim 1, characterized in that, The scheduling process supports a hybrid scheduling of GPU-accelerated and CPU-intensive tasks.

6. The heterogeneous database snapshot comparison method according to claim 1, characterized in that, The method further includes: A reinforcement learning model is used to optimize the data comparison strategy. The objective function is to minimize the comparison time and false alarm rate. The state space includes the block change probability, transaction latency and network bandwidth. The action space includes adjusting the comparison time window and merging adjacent blocks. The reward function is a weighted value of the comparison accuracy and resource consumption.

7. A heterogeneous database snapshot comparison device, characterized in that, include: The mapping rule generation module is used to generate cross-database mapping rules between the source database and the target database using a language model. The comparison task scheduling module is used to build a distributed comparison task pool based on Kubernetes and to schedule comparison tasks in parallel according to data blocks. The comparison task execution module is used to execute comparison tasks, and reads the corresponding partition data from the source database and the target database for comparison based on the cross-database mapping rules.

8. A computer device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the heterogeneous database data snapshot comparison method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the heterogeneous database snapshot comparison method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the heterogeneous database data snapshot comparison method as described in any one of claims 1-6.

Images