Data comparison method and device, computer device, readable storage medium and program product

Abstract

The application relates to a data comparison method and device, computer equipment, a computer readable storage medium and a computer program product, relates to the technical field of databases, and can ensure normal business during data comparison. The method comprises the following steps: determining to-be-compared data; when the to-be-compared data comprises a data table of a target end database, determining a first to-be-compared sharding table in a plurality of sharding tables contained in the data table; the data table is obtained by performing data migration on sharding tables respectively provided by a plurality of database nodes in a distributed source end database, and the first to-be-compared sharding table is part of the plurality of sharding tables; a second sharding table is acquired in a database node of the distributed source end database providing the first sharding table, and the second sharding table and the first sharding table are compared.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and in particular, to a data comparison method and device, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] With the development and update of database products and the change of user usage requirements, there will be a continuous demand for data migration. In a scenario with high requirements for data accuracy, after migrating database objects and data from a source database to a target database, users often want to verify the accuracy of this migration. As a result, there is a demand for data comparison before and after migration.

[0003] For many-to-one database migration, that is, migrating data of multiple database nodes of a distributed database to another target database, related technologies mainly directly extract data of each database node below and compare it with data obtained by migration of the target database.

[0004] However, the inventors have found in practice that when using the above method for data comparison, other businesses of the distributed source database often cannot be normally processed, and there is a problem of data comparison hindering normal performance of other businesses. SUMMARY

[0005] Therefore, it is necessary to provide a data comparison method, device, computer device, computer readable storage medium and computer program product to solve the above technical problems.

[0006] In a first aspect, the present application provides a data comparison method, comprising:

[0007] determining to-be-compared data;

[0008] when the to-be-compared data includes a data table of a target end database, determining a first sharded table to be currently compared in a plurality of sharded tables contained in the data table; the data table is obtained by data migration of a plurality of sharded tables respectively provided by a plurality of database nodes in a distributed source end database, and the first sharded table to be currently compared is part of the plurality of sharded tables;

[0009] obtaining a second sharded table in a database node of the distributed source end database providing the first sharded table, and comparing the second sharded table and the first sharded table.

[0010] In one embodiment, the determining of the first sharded table to be currently compared in the plurality of sharded tables contained in the data table comprises:

[0011] In the plurality of database nodes of the distributed source-side database, a target database node to be compared with data is determined, and source information of a sharded table is determined according to the target database node; the target database node is part of the plurality of database nodes;

[0012] According to the source information of the sharded table, a plurality of sharded tables contained in the data table are filtered to obtain a sharded table from the target database node as a first sharded table.

[0013] In one embodiment, the determination of the target database node to be compared with data in the plurality of database nodes of the distributed source-side database comprises:

[0014] In response to a database node selection instruction triggered by a user, a database node selected by the user is determined as the target database node in the plurality of database nodes of the distributed source-side database;

[0015] Alternatively,

[0016] The plurality of data pulling time periods include different data pulling time periods.

[0017] For each data pulling time period, when the current time matches the data pulling time period, the database node associated with the data pulling time period is determined as the target database node.

[0018] In one embodiment, the data table includes a data source field.

[0019] According to the source information of the sharded table, a plurality of sharded tables contained in the data table are filtered to obtain a sharded table from the target database node as a first sharded table.

[0020] For each sharded table contained in the data table, field information of the sharded table under the data source field is determined.

[0021] The field information of each sharded table is matched with the source information of the sharded table, and a sharded table from the target database node is obtained according to the sharded table corresponding to the matched field information.

[0022] In one embodiment, before the step of determining data to be compared, the method further comprises:

[0023] In the data migration stage, a data source identifier corresponding to each database node is determined; the data source identifier is used to indicate that the data source is the corresponding database node.

[0024] According to the shard tables provided by the plurality of database nodes, a data table containing a data source field is generated, and the data source identifier of each database node providing the shard table is identified as field information of the data source field of the corresponding shard table.

[0025] In one embodiment, the determination of the data source identifier corresponding to each database node comprises:

[0026] The IP address and port information corresponding to each database node are determined.

[0027] For each database node, the data source identifier corresponding to the database node is obtained according to the splicing result of the IP address and port information corresponding to the database node.

[0028] In a second aspect, the present application further provides a data comparison device, comprising:

[0029] A data acquisition module is configured to determine to-be-compared data.

[0030] A shard acquisition module is configured to, when the to-be-compared data includes a data table of a target end database, determine a first shard table to be currently compared in a plurality of shard tables contained in the data table; the data table is obtained by performing data migration on shard tables respectively provided by a plurality of database nodes in a distributed source end database, and the first shard table to be currently compared is part of the plurality of shard tables.

[0031] A shard comparison module is configured to acquire a second shard table in a database node of the distributed source end database providing the first shard table, and compare the second shard table and the first shard table.

[0032] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0033] To-be-compared data is determined.

[0034] When the to-be-compared data includes a data table of a target end database, a first shard table to be currently compared is determined in a plurality of shard tables contained in the data table; the data table is obtained by performing data migration on shard tables respectively provided by a plurality of database nodes in a distributed source end database, and the first shard table to be currently compared is part of the plurality of shard tables.

[0035] A second shard table is acquired in a database node of the distributed source end database providing the first shard table, and the second shard table and the first shard table are compared.

[0036] In a fourth aspect, the present application further provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the following steps:

[0037] determining data to be compared;

[0038] when the data to be compared comprises a data table of a target end database, determining a first shard table to be compared currently among a plurality of shard tables contained in the data table, the data table being obtained by data migration on shard tables respectively provided by a plurality of database nodes in a distributed source end database, the first shard table to be compared currently being part of the plurality of shard tables;

[0039] acquiring a second shard table in a database node of the distributed source end database providing the first shard table, and comparing the second shard table and the first shard table.

[0040] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0041] determining data to be compared;

[0042] when the data to be compared comprises a data table of a target end database, determining a first shard table to be compared currently among a plurality of shard tables contained in the data table, the data table being obtained by data migration on shard tables respectively provided by a plurality of database nodes in a distributed source end database, the first shard table to be compared currently being part of the plurality of shard tables;

[0043] acquiring a second shard table in a database node of the distributed source end database providing the first shard table, and comparing the second shard table and the first shard table.

[0044] The data comparison method, device, computer device, computer readable storage medium, and computer program product, the server can determine the data to be compared. When the data to be compared includes a data table of the target end database, the first shard table to be compared currently can be determined in a plurality of shard tables included in the data table. The data table is obtained by performing data migration on a plurality of shard tables respectively provided by a plurality of database nodes in the distributed source end database. The first shard table to be compared currently is part of the plurality of shard tables. Then, the second shard table is obtained in the database node of the distributed source end database providing the first shard table, and the second shard table and the first shard table are compared. In the embodiment of the present application, when the data to be compared includes the data table of the target end database, part of the shard tables in the data table is determined as the first shard table to be compared, and then the second shard table is pulled from the corresponding database node for comparison. According to the correspondence between the shard table and the database node, the data comparison is performed for each shard data of the distributed database node in time periods, the content of the data comparison task is avoided to involve all the database node data sets in the same time period, the excessive pressure on all nodes is avoided, and the influence on the production business of the distributed source end database is effectively reduced. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 A flowchart of a data comparison method in an embodiment;

[0047] Figure 2 A schematic diagram of a data migration process in an embodiment;

[0048] Figure 3 A flowchart of a step of determining a first shard table in an embodiment;

[0049] Figure 4 A structural block diagram of a data comparison device in an embodiment;

[0050] Figure 5 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0052] In order to make the present application better understood by those skilled in the art, the related art will be introduced first.

[0053] With the development and update of database products and the change of user usage requirements, there will be a continuous demand for data migration. Users can realize the migration of data between different databases through a data migration tool. Due to the complexity of the database system, the data migration work involves many details. In a scenario with high requirements for data accuracy, after migrating database objects and data from a source database to a target database, users often want to verify the accuracy of this migration. As a result, there is a need for data comparison before and after migration.

[0054] In the related art, the function of checking data before and after migration provided by the data migration tool is basically for one-to-one database topology, that is, the source database is a single database instance, and the target database is also a single database instance, for example, data in a MySQL type database is migrated to another MySQL type database, or data in a PostgreSQL type database is migrated to another PostgreSQL type database. For many-to-one database migration, such as migrating data in a distributed database to a PostgreSQL type database, wherein the distributed database as the source database has multiple database instances (also referred to as nodes of the distributed database). In the face of such a topology, the current data migration tool data comparison adopts a simplified one-to-one topology, and the working process is as follows: for the source database, the entire source database is considered as a single database instance, and the data of each database node under the source database proxy layer is directly extracted and compared with the data obtained by migrating the target database.

[0055] This way is simple and intuitive, but has some drawbacks. On the one hand, the distributed database often stores a large amount of data, and the above-mentioned method needs to extract a large amount of data from the distributed database when performing data comparison. On the other hand, when extracting the data of each database node, all database nodes of the distributed database consume resources for data comparison during the data comparison period, which seriously hinders normal production business. It can be seen that the related art has the problem of low data comparison efficiency.

[0056] Therefore, it is necessary to provide a data comparison method, device, computer equipment, computer readable storage medium and computer program product to solve the above technical problems.

[0057] In one embodiment, as shown in Figure 1 A data comparison method is provided, and the embodiment is exemplified by the method applied to a server. It can be understood that the method can also be applied to a system including a terminal and a server, and implemented through the interaction of the terminal and the server. The server can be connected to a target end database and a distributed source end database through a network.

[0058] In the embodiment, the method includes the following steps:

[0059] S101, determining to-be-compared data.

[0060] The to-be-compared data is two groups of data to be compared. In the embodiment, the to-be-compared data can include data in the source end database before data migration and data in the target end database after data migration. In the data migration process, the data sending end is referred to as the source end database, and the data receiving end is referred to as the target end database. Data migration can be understood as copying the structure and data in one database from one database system to another database system. Structure migration, that is, migrating the object structure in the database, such as table, view, function, and stored procedure. Data comparison can be understood as comparing whether the data of the source end database and the target end database involved in the data migration process is consistent.

[0061] In some embodiments, data migration can include one or more of full migration and incremental synchronization. Full migration refers to migrating offline data of the source end database, and incremental synchronization refers to migrating real-time data of the source end database.

[0062] S102, when the to-be-compared data includes a data table of the target end database, determining a first to-be-compared shard table in a plurality of shard tables included in the data table. The data table is obtained by data migration on a plurality of shard tables respectively provided by a plurality of database nodes in the distributed source end database, and the first to-be-compared shard table is part of the plurality of shard tables.

[0063] In actual application, the to-be-compared data can include shard tables in the distributed source end database and data tables in the target end database. The data tables in the target end database can be obtained by combining the shard tables provided by the plurality of database nodes in the distributed source end database, for example, by combining information in the shard tables provided by part or all of the database nodes. The data packet can also be referred to as a logical table.

[0064] The distributed source-end database can have one or more corresponding distributed database middleware. The distributed database middleware is a database middleware for solving the performance degradation caused by excessive data volume in a single machine database. The distributed database middleware can perform database and table splitting, and store data on multiple database instances (also referred to as database nodes) in the underlying distributed database. Each database node can store one or more shard tables. In some embodiments, the distributed database middleware can have a corresponding distributed database middleware agent layer, which is an application-oriented agent program commonly provided by the distributed database middleware. The agent program can be used to manage metadata and data routing of database and table splitting.

[0065] A shard table is a physical table stored by a database node in a distributed database. In subsequent data migration, the table information in the shard table becomes part of a logical table. In some examples, a physical table in a database node can be understood as a shard table, and all related physical table shards form a logical table. Therefore, a physical table used to form a logical table (or a database in a target-end database) is also referred to as a shard table.

[0066] For example, referring to Figure 2 For a distributed database, the application sees a logical table name from the agent layer, such as SchemaA.TableA. However, there is no table SchemaA.TableA in each database node underlying the distributed database. The actual situation can be that the logical table SchemaA.TableA is composed of physical tables SchemaA_1.TableA_1, SchemaA_1. TableA_2, SchemaA_2. TableA_1, and SchemaA_2. TableA_2, where the physical tables SchemaA_1. TableA_1 and SchemaA_1. TableA_2 are in the schema SchemaA_1 of database node 1, and the physical tables SchemaA_2. TableA_1 and SchemaA_2. TableA_2 are in the schema SchemaA_2 of database node 2.

[0067] Of course, in actual applications, there can be other forms of data tables, such as single-table and global tables. A single-table is a logical table composed of only one physical table shard, that is, the physical table information forming the logical table exists in a single node of the distributed database, and the logical table name is the physical table name. The logical table name of a global table is the physical table name, and the same table structure and data exist in each node of the distributed database.

[0068] When it is determined that the data to be compared includes a data table of the target end database, a first shard table to be currently compared can be determined in the plurality of shard tables included in the data table.

[0069] S103, obtaining a second shard table in the database node providing the first shard table of the distributed source end database, and comparing the second shard table and the first shard table.

[0070] Compared with the related art, when the many-to-one data migration is compared, the topology structure inside the distributed database is not concerned and not known, and only the distributed source end database is considered as a single database instance. In this step, when it is determined that the data to be compared includes a data table composed of a plurality of shard tables, and the first shard table to be currently compared is determined, the database node providing the first shard table can be determined in the distributed source end database according to the correspondence between the shard tables in the data table and each database node, and then the corresponding shard table, i.e. the second shard table, is obtained from the database node, and then the first shard table and the second shard table are compared.

[0071] It should be noted that, for the related art, the data of each database node below the proxy layer is directly extracted and compared with the data extracted from the target database. The inventor found that this processing method would cause a large amount of data to be extracted from the distributed source end database and all database nodes to consume resources for data comparison, thereby seriously hindering normal production business. In this embodiment, by obtaining part of the plurality of shard tables in the data table, i.e. the first shard table, on the one hand, the plurality of shard tables contained in each can be compared in different time periods, reducing the data comparison pressure of different database nodes, and on the other hand, the data amount of pulling data from the distributed source end database can be reduced, thereby effectively avoiding hindering normal production business, and improving the data comparison efficiency.

[0072] In the data comparison method, the server can determine the data to be compared. When the data to be compared includes a data table of the target end database, a first shard table to be currently compared can be determined in a plurality of shard tables included in the data table. The data table is obtained by performing data migration on a plurality of shard tables respectively provided by a plurality of database nodes of the distributed source end database. The first shard table to be currently compared is part of the plurality of shard tables. Then, a second shard table is obtained from a database node of the distributed source end database providing the first shard table, and the second shard table and the first shard table are compared. In the embodiment of the present application, when the data to be compared includes the data table of the target end database, part of the shard tables in the data table is determined as the first shard table to be compared, and then the second shard table is pulled from the corresponding database node for comparison. According to the correspondence between the shard table and the database node, the data comparison is performed for each shard data of the distributed database node in time periods, so as to avoid involving all database nodes in the data comparison task in the same time period, and avoid causing excessive pressure on all nodes, thereby effectively reducing the influence on the production business of the distributed source end database.

[0073] In one embodiment, as shown in FIG. 10, in step S102, determining the first shard table to be currently compared in the plurality of shard tables included in the data table can include the following steps: Figure 3

[0074] S301, in the plurality of database nodes of the distributed source end database, a target database node to be compared is determined, and shard table source information is determined according to the target database node. The target database node is part of the plurality of database nodes.

[0075] In the specific implementation, when the distributed source end database is compared with the target end database obtained after migration, the target database node to be compared can be determined in the plurality of database nodes of the distributed source end database. The target database node is one or more, and the target database node can be manually selected by a user or can be filtered by the server according to node selection logic.

[0076] After the target database node is determined, the data provided by the target database node and the data opposite to the data in the data table can be compared, so the shard table source information can be determined according to the target database node. The shard table source information can be information representing the source of the shard table data, for example, if the target database node is node A, the shard table source information can be information indicating that the shard table comes from node A.

[0077] S302, according to the shard table source information, filtering a plurality of shard tables included in the data table to obtain a shard table from the target database node as a first shard table.​

[0078] Further, the plurality of shard tables contained in the data table can be filtered by using the shard table source information, and the filtered shard table from the target database node is taken as the first shard table.

[0079] In the embodiment, by taking part of the plurality of database nodes as the target database node, and then screening the first shard table according to the target database node, it can avoid data pulling and data comparison of all database nodes of the distributed source database at the same time, effectively reduce the computing pressure of the database node, avoid consuming a large amount of resources for data comparison, and help to ensure that other businesses of the distributed source database can be normally performed.

[0080] In one embodiment, in step S301, in the plurality of database nodes of the distributed source database, the target database node to be compared is determined, which can include the following steps:

[0081] In response to a database node selection instruction triggered by a user, in the plurality of database nodes of the distributed source database, the database node selected by the user is determined as the target database node; or, the data pulling time period of each database node of the distributed source database is determined; the plurality of data pulling time periods contain different data pulling time periods; for each data pulling time period, when the current time matches the data pulling time period, the database node associated with the data pulling time period is taken as the target database node.

[0082] In actual application, the database node selection instruction for selecting the database node can be triggered by the user, and the server determines the database node selected by the user as the target database node in the plurality of database nodes of the distributed source database. In some embodiments, the database node selected by the user does not exceed a node quantity threshold. The node quantity threshold can be determined according to the data processing capacity of the distributed source database, for example, the node quantity threshold can be determined according to the total amount of idle computing resources of the distributed source database. The node quantity threshold is positively correlated with the total amount of idle computing resources, so that when the distributed source database has more idle computing resources, more database nodes can be allowed to participate in the data comparison process, and when the idle computing resources of the distributed source database are less, the database nodes participating in the data comparison are reduced, which can ensure that enough database nodes are reserved to participate in normal production business.

[0083] In some other embodiments, the server can also be provided in advance with the data pulling time period configuration of the database nodes, in which the respective data pulling time periods of the plurality of database nodes in the distributed source database can be recorded, wherein the plurality of data pulling time periods can include different data pulling time periods, that is, in the plurality of data pulling time periods, some or all of the data pulling time periods are different. Then, for each data pulling time period, when the current time matches the data pulling time period, the database node associated with the data pulling time period can be taken as the target database node.

[0084] In the embodiment, the user can flexibly select part of the database nodes as the target database nodes, or part of the database nodes can be taken as the target database nodes according to the data pulling time periods, so that all the database nodes in the distributed source database can be effectively avoided from participating in the data comparison, and the normal operation of other businesses of the distributed source database can be ensured.

[0085] In one embodiment, the data table includes a data source field, for example, as shown in Figure 2 The data table in the target database can include a data source field "TAG", and the field information under the field can represent the data source of the shard table, that is, the database node providing the shard table.

[0086] Correspondingly, in step S302, the plurality of shard tables contained in the data table are filtered according to the shard table source information to obtain the shard tables from the target database nodes, which can include the following steps:

[0087] For each shard table contained in the data table, the field information of the shard table under the data source field is determined; the field information of each shard table is matched with the shard table source information, and the shard table corresponding to the matching successful field information is taken as the shard table from the target database node.

[0088] In actual application, when the data in the distributed source database is migrated to the target database and the corresponding data table is generated, the data source field can be set in the data table to record the source of each shard table, so that in the determination of the shard table source information, for each shard table contained in the data table, the field information of the shard table under the data source field can be read, and then the field information of each shard table is matched with the shard table source information, and the shard table corresponding to the matching successful field information is determined as the shard table from the target database node.

[0089] In the embodiment, the field information under the data source field in the data table can be used to quickly screen the shard table from the target database node from the data table, so that the number of shard tables for comparison can be effectively reduced, and the shard table related to the current data comparison can be accurately determined.

[0090] In one embodiment, before step S101, the method can further include the following steps:

[0091] In the data migration stage, the data source identifier corresponding to each database node is determined; a data table containing a data source field is generated according to the shard tables provided by the plurality of database nodes, and the data source identifier of each database node providing a shard table is added as field information of the data source field of the corresponding shard table.

[0092] In a specific implementation, in the data migration stage, the data source identifier corresponding to each database node can be determined in advance, where the data source identifier is used to indicate that the data source is the corresponding database node, and the data source identifier can have uniqueness, so that the data source identifiers of different database nodes can be distinguished, so as to subsequently indicate the database node providing a shard table through the data source identifier.

[0093] Then, when the logical table is determined according to the shard tables provided by the plurality of database nodes and migrated to the target end database to obtain the corresponding data table, the data source field can be added in the data table, and the data source identifier of each database node providing a shard table is added to the data source field of the corresponding shard table of the data table as field information of the data source field.

[0094] In this embodiment, in the data migration stage, an additional field is added to the table object migrated to the target end database to record the source node of each piece of data, and the data source identifier is added as field information of the data source field of the corresponding shard table. Through the field, the part of data corresponding to a certain shard of the source end migrated to the target end database can be defined on the target end database, so that the shard comparison of the shard table data is realized, and a basis is provided for subsequent quick determination of a specific data shard and determination of a related database node.

[0095] In one embodiment, determining the data source identifier corresponding to each database node can include the following steps:

[0096] The IP address and port information corresponding to each database node are determined; and for each database node, the data source identifier corresponding to the database node is obtained according to the splicing result of the IP address and port information corresponding to the database node.

[0097] In a specific implementation, for each database node, the IP address and port information corresponding to the database node can be obtained, and then the data source identifier corresponding to the database node is obtained according to the splicing result of the IP address and port information, so that the data source identifier is quickly obtained, and the correspondence between the data source identifier and the database node is effectively ensured, so as to facilitate the shard table distinction and screening through the data source identifier.

[0098] In order for those skilled in the art to better understand the above steps, the embodiments of the present application are exemplarily illustrated by an example below, but it should be understood that the embodiments of the present application are not limited thereto.

[0099] Reference Figure 2 In actual application, a user uses a data migration system to perform data migration, and before performing data migration, information of several steps involved in a database migration task can be configured on a system console, wherein a configuration item about comparison of sharding data after data migration of a distributed database middleware is involved. The following are steps of configuring a task, including:

[0100] (1) The user fills in connection information of a source database and a target database, and the data migration system can identify whether the source database is a distributed database and identify a specific node topology structure under the distributed database.

[0101] (2) The user selects a migration stage required for the migration task: structure migration, full migration, and incremental synchronization. The user can select any combination of structure migration, full migration, and incremental synchronization. In an example, if the user does not select the structure migration stage, the task does not support comparison of sharding data of the distributed database, and a traditional full data comparison mode will be used to compare data on all nodes subsequently.

[0102] (3) The user selects a source database object that wants to migrate to the target database, and the data migration system can identify whether the selected source database table object is a sharding table, a single-table, or a global table.

[0103] (4) The data migration system can provide the following configuration for each selected source database sharding table: in the structure migration stage, whether to add a table field (i.e., a data source field) to record node information of the source distributed database of the data row of the target database on the basis of the table structure of the source database when migrating the table structure of the source sharding table to the target. Figure 2 The data source field "TAG" in the example is not added with the prefix. Thus, the task configuration is completed.

[0104] After the task configuration is completed, the user can start the task, and the following are related steps in a task running stage, including:

[0105] (1) Structure migration stage: when migrating the table structure of the source sharding table to the target, according to the configuration information of the task, the migration thread adds a table field TAG on the basis of the table structure of the source database.

[0106] (2) Full migration stage: This stage is an optional stage when configuring the task. If full migration is configured, the data migration worker pulls the data from the source database, and generates INSERT SQL based on the original data, and adds the insertion value of the TAG field created by the data migration system for the target table structure. The value of the TAG field is the source node information of the data in the source database, which can be the information obtained by combining "IP+PORT information", or other database node identification information.

[0107] (3) Incremental synchronization stage: This stage is an optional stage when configuring the task. If incremental synchronization is configured, the data migration worker pulls the transaction log from the source database and parses the original semantics. If it is an INSERT type DML log event, the data migration system generates INSERT SQL based on the original value, and adds the insertion value of the TAG field created for the target table structure.

[0108] After the task runs to the incremental synchronization stage, or the task has no incremental synchronization stage and the full migration stage has been completed, the user can create a data comparison task as needed.

[0109] During the running of the data comparison task, the user can create a data comparison task and check the source table object and its corresponding target table object that the user wants to compare. For the table object to be compared, the user can be prompted to choose whether to use the traditional full data comparison method or the sharded data comparison method according to whether the source table object is a sharded table and whether the corresponding target table object has a TAG field created additionally. If the user chooses to use the sharded data comparison method, the user is further prompted to choose which node of the source distributed database the sharded data to be compared is in.

[0110] According to the user's selected parameters, if the sharded data comparison method is used, for a sharded table, the data comparison thread pulls the sharded data of the table object involved on the node of the source distributed database specified by the user, and at the same time, when pulling the data on the corresponding table object in the target database, the SQL filtering condition is used to filter the data from the source node with the TAG field. Thus, it can be ensured that the data ranges compared on both sides are consistent. For the data pulled from the source and the target, the subsequent data can be sorted by the primary key or unique key of the table, thereby ensuring that the data compared on both sides are one-to-one corresponding in order. After the data range and data order are determined, the two parts of data can be compared and the comparison result can be recorded.

[0111] Compared with the traditional data comparison method of the distributed database middleware, the embodiment of the application can compare the data of the distributed database middleware in different time periods, avoid the data comparison task involving all the data sets of the database nodes in the same time period, and reduce the impact on the production business.

[0112] It should be understood that, although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0113] Based on the same inventive concept, the embodiment of the application also provides a data comparison device for implementing the above-mentioned data comparison method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data comparison device embodiments provided below can refer to the limitations of the data comparison method in the above text, and will not be repeated here.

[0114] In one exemplary embodiment, as shown in Figure 4 a data comparison device is provided, comprising:

[0115] The data acquisition module 401 is configured to determine the data to be compared.

[0116] The shard acquisition module 402 is configured to determine a first shard table to be compared currently in a plurality of shard tables contained in a data table of a target end database when the data to be compared includes the data table, the data table being obtained by data migration on a plurality of shard tables respectively provided by a plurality of database nodes of a distributed source end database, and the first shard table to be compared currently being part of the plurality of shard tables.

[0117] The shard comparison module 403 is configured to acquire a second shard table in a database node of the distributed source end database providing the first shard table, and compare the second shard table and the first shard table.

[0118] In one embodiment, the shard acquisition module 402 is configured to:

[0119] The target database node to be compared with data is determined in the plurality of database nodes of the distributed source-end database, and the shard table source information is determined according to the target database node; the target database node is part of the plurality of database nodes;

[0120] According to the shard table source information, a plurality of shard tables contained in the data table are filtered to obtain a shard table from the target database node as a first shard table.

[0121] In one embodiment, the shard obtaining module 402 is configured to:

[0122] In response to a database node selection instruction triggered by a user, a database node selected by the user is determined as a target database node in the plurality of database nodes of the distributed source-end database;

[0123] Alternatively,

[0124] The data pulling time periods of the plurality of database nodes of the distributed source-end database are determined; the plurality of data pulling time periods include different data pulling time periods;

[0125] For each data pulling time period, when the current time matches the data pulling time period, a database node associated with the data pulling time period is determined as a target database node.

[0126] In one embodiment, the data table includes a data source field;

[0127] The shard obtaining module 402 is configured to:

[0128] For each shard table contained in the data table, field information of the shard table under the data source field is determined;

[0129] The field information of each shard table is matched with the shard table source information, and a shard table from the target database node is obtained according to a shard table corresponding to field information that is matched successfully.

[0130] In one embodiment, the apparatus further includes a field information adding module, and the field information adding module is configured to:

[0131] In the data migration stage, a data source identifier corresponding to each database node is determined; the data source identifier is used to indicate that the data source is the corresponding database node;

[0132] According to a plurality of shard tables provided by the plurality of database nodes, a data table including a data source field is generated, and the data source identifier of each database node providing the shard table is used as field information of the data source field of the corresponding shard table.

[0133] In one embodiment, the field information adding module is configured to:

[0134] determine the IP address and port information corresponding to each of the database nodes;

[0135] for each of the database nodes, obtain the data source identifier corresponding to the database node according to a splicing result of the IP address and port information corresponding to the database node.

[0136] The modules in the data comparison device can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by the processor to perform the operations of the modules.

[0137] In one example embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 5 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data of a target end database. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a data comparison method.

[0138] Those skilled in the art can understand that Figure 5 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0140] In an embodiment, a computer readable storage medium is provided, and the computer readable storage medium has stored thereon a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0141] In an embodiment, a computer program product is provided, and the computer program product includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0142] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the related data need to comply with relevant regulations.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0144] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0145] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data comparison method, characterized in that, The method includes: Identify the data to be compared; When the data to be compared includes a data table in the target database, the multiple shard tables contained in the data table are identified as corresponding to multiple database nodes in the distributed source database. The target database node is determined from the multiple database nodes, and the first shard table to be compared is obtained from the data table based on the target database node. The data table is obtained by migrating data from the shard tables provided by the multiple database nodes in the distributed source database. The first shard table to be compared is a part of the multiple shard tables, and the target database node is a part of the multiple database nodes. The number of target database nodes does not exceed a node number threshold, which is determined based on the total amount of idle computing resources in the distributed source database. A second shard table is obtained from the target database node that provides the first shard table in the distributed source database, and the second shard table and the first shard table are compared. During the comparison between the second shard table and the first shard table, other first shard tables in the data table are not compared with other corresponding second shard tables in the distributed source database at the same time. The other first shard tables are shard tables other than the first shard table, and the other second shard tables are shard tables other than the second shard table.

2. The method according to claim 1, characterized in that, The step of obtaining the first shard table to be compared from the data table based on the target database node includes: Determine the source information of the sharded table based on the target database node; Based on the sharding table source information, multiple sharding tables contained in the data table are filtered to obtain the sharding table from the target database node, which is used as the first sharding table.

3. The method according to claim 2, characterized in that, Determining the target database node from the plurality of database nodes includes: In response to a user-triggered database node selection command, the database node selected by the user is determined as the target database node among the multiple database nodes of the distributed source database. or, Determine the data retrieval time period for each of the plurality of database nodes in the distributed source database; the plurality of data retrieval time periods include different data retrieval time periods; For each data retrieval period, when the current time matches the data retrieval period, the database node associated with the data retrieval period is taken as the target database node.

4. The method according to claim 2, characterized in that, The data table includes a data source field; The step of filtering multiple shard tables contained in the data table based on the shard table source information to obtain the shard table from the target database node includes: For each shard table contained in the data table, determine the field information of the shard table under the data source field; The field information of each shard table is matched with the source information of the shard table, and the shard table from the target database node is obtained based on the shard table corresponding to the successfully matched field information.

5. The method according to claim 4, characterized in that, Before the step of determining the data to be compared, the method further includes: During the data migration phase, a data source identifier is determined for each database node; the data source identifier is used to indicate that the data source is the corresponding database node. Based on the sharded tables provided by the multiple database nodes, a data table containing a data source field is generated, and the data source identifier of each database node that provides the sharded table is used as the field information of the data source field of the corresponding sharded table.

6. The method according to claim 5, characterized in that, Determining the data source identifier corresponding to each database node includes: Determine the IP address and port information corresponding to each of the database nodes; For each database node, the data source identifier corresponding to the database node is obtained by concatenating the IP address and port information corresponding to the database node.

7. A data comparison device, characterized in that, The device includes: The data acquisition module is used to determine the data to be compared. The shard acquisition module is used to determine, when the data to be compared includes a data table in the target database, the multiple shard tables contained in the data table and the multiple database nodes corresponding to them in the distributed source database, determine the target database node from the multiple database nodes, and obtain the first shard table to be compared from the data table based on the target database node; the data table is obtained by migrating data from the shard tables provided by the multiple database nodes in the distributed source database, the first shard table to be compared is a part of the multiple shard tables, the target database node is a part of the multiple database nodes, the number of the target database nodes does not exceed a node number threshold, and the node number threshold is determined based on the total amount of idle computing resources of the distributed source database; The shard comparison module is used to obtain a second shard table from the target database node where the first shard table is provided by the distributed source database, and to compare the second shard table with the first shard table. During the comparison process between the second shard table and the first shard table, other first shard tables in the data table are not compared with other corresponding second shard tables in the distributed source database at the same time. The other first shard tables are shard tables other than the first shard table, and the other second shard tables are shard tables other than the second shard table.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images