Data comparison method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data comparison method and device, computer equipment, a computer readable storage medium and a computer program product, relates to the technical field of databases, and can ensure normal operation of businesses during data comparison. The method comprises the steps of determining to-be-compared data; when the to-be-compared data comprises a data table of a target end database, determining a first fragment table to be compared at present in a plurality of fragment tables contained in the data table; the data table is obtained by performing data migration on fragmentation tables provided by a plurality of database nodes in a distributed source end database, and the first fragmentation table to be compared currently is part of the plurality of fragmentation tables; and acquiring a second fragmentation table from a database node, which provides the first fragmentation table, of the distributed source end database, and comparing the second fragmentation table with the first fragmentation table.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data comparison method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development and update of database products and the changes in user needs, there will be a continuous demand for data migration. In scenarios with high requirements for data accuracy, after migrating database objects and data from the source database to the target database, users often want to verify the accuracy of the migration work, which leads to the need to compare the data before and after the migration.

[0003] For many-to-one database migration, that is, migrating the data of multiple database nodes of a distributed database to another target database, the relevant technology mainly directly extracts the data of each underlying database node during data comparison and compares it with the data obtained by migrating the target database.

[0004] However, the inventors have found in practice that when using the above method to perform data comparison, other businesses of the distributed source database often cannot be processed normally, and there is a problem that data comparison hinders the normal progress of other businesses. Summary of the invention

[0005] Based on this, it is necessary to provide a data comparison method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a data comparison method, comprising:

[0007] Determine the data to be compared;

[0008] When the data to be compared includes a data table of a target database, a first shard table to be compared is determined from among the multiple shard tables included in the data table; the data table is obtained by performing data migration on shard tables provided by multiple database nodes in a distributed source database, and the first shard table to be compared is part of the multiple shard tables;

[0009] A second shard table is obtained from a database node of the distributed source database that provides the first shard table, and the second shard table is compared with the first shard table.

[0010] In one embodiment, determining a first shard table to be compared among the multiple shard tables included in the data table includes:

[0011] Determine a target database node to be compared with data among the multiple database nodes of the distributed source database, and determine the shard table source information according to the target database node; the target database node is part of the multiple database nodes;

[0012] According to the shard table source information, multiple shard tables included in the data table are filtered to obtain a shard table from the target database node as the first shard table.

[0013] In one embodiment, determining a target database node to be compared with data among the multiple database nodes of the distributed source database includes:

[0014] In response to a database node selection instruction triggered by a user, determining a database node selected by the user as a target database node among the multiple database nodes of the distributed source database;

[0015] or,

[0016] Determine a data pulling period for each of the plurality of database nodes of the distributed source database; the plurality of data pulling periods include different data pulling periods;

[0017] For each of the data pulling periods, when the current time matches the data pulling period, the database node associated with the data pulling period is used as the target database node.

[0018] In one of the embodiments, the data table includes a data source field;

[0019] The filtering of the multiple shard tables contained in the data table according to the shard table source information to obtain the shard table from the target database node includes:

[0020] For each of the shard tables included in the data table, determining field information of the shard table under the data source field;

[0021] The field information of each shard table is matched with the shard table source information, and the shard table from the target database node is obtained according to the shard table corresponding to the successfully matched field information.

[0022] In one embodiment, before the step of determining the data to be compared, the method further includes:

[0023] In the data migration phase, a data source identifier corresponding to each database node is determined; the data source identifier is used to indicate that the data source is the corresponding database node;

[0024] A data table including a data source field is generated according to the shard tables provided by the plurality of database nodes, and the data source identifier of each database node providing the shard table is used as field information of the data source field of the corresponding shard table.

[0025] In one embodiment, determining the data source identifier corresponding to each database node includes:

[0026] Determine the IP address and port information corresponding to each of the database nodes;

[0027] For each of the database nodes, a data source identifier corresponding to the database node is obtained based on the splicing result of the IP address and port information corresponding to the database node.

[0028] In a second aspect, the present application also provides a data comparison device, comprising:

[0029] A data acquisition module, used to determine the data to be compared;

[0030] A shard acquisition module is used to determine a first shard table to be compared from among multiple shard tables contained in the data table when the data to be compared includes a data table of a target database; the data table is obtained by performing data migration on shard tables provided by multiple database nodes in a distributed source database, and the first shard table to be compared is part of the multiple shard tables;

[0031] The shard comparison module is used to obtain a second shard table from the database node that provides the first shard table in the distributed source database, and compare the second shard table with the first shard table.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Determine the data to be compared;

[0034] When the data to be compared includes a data table of a target database, a first shard table to be compared is determined from among the multiple shard tables included in the data table; the data table is obtained by performing data migration on shard tables provided by multiple database nodes in a distributed source database, and the first shard table to be compared is part of the multiple shard tables;

[0035] A second shard table is obtained from a database node of the distributed source database that provides the first shard table, and the second shard table is compared with the first shard table.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0037] Determine the data to be compared;

[0038] When the data to be compared includes a data table of a target database, a first shard table to be compared is determined from among the multiple shard tables included in the data table; the data table is obtained by performing data migration on shard tables provided by multiple database nodes in a distributed source database, and the first shard table to be compared is part of the multiple shard tables;

[0039] A second shard table is obtained from a database node of the distributed source database that provides the first shard table, and the second shard table is compared with the first shard table.

[0040] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0041] Determine the data to be compared;

[0042] When the data to be compared includes a data table of a target database, a first shard table to be compared is determined from among the multiple shard tables included in the data table; the data table is obtained by performing data migration on shard tables provided by multiple database nodes in a distributed source database, and the first shard table to be compared is part of the multiple shard tables;

[0043] A second shard table is obtained from a database node of the distributed source database that provides the first shard table, and the second shard table is compared with the first shard table.

[0044] The above-mentioned data comparison method, device, computer equipment, computer-readable storage medium and computer program product, the server can determine the data to be compared, when the data to be compared includes a data table of the target database, can determine the first shard table to be compared from the multiple shard tables contained in the data table, wherein the data table is obtained by migrating data from the shard tables provided by multiple database nodes in the distributed source database, and the first shard table to be compared is a part of the multiple shard tables; then, the second shard table is obtained from the database node that provides the first shard table in the distributed source database, and the second shard table is compared with the first shard table. In the embodiment of the present application, when the data to be compared includes the data table of the target database, some shard tables in the data table are determined as the first shard tables to be compared, and then the second shard tables are pulled from the corresponding database nodes for comparison. According to the correspondence between the shard tables and the database nodes, data comparison can be performed in different time periods for each shard data of the distributed database nodes, thereby avoiding the content of the data comparison task involving the full set of data of all database nodes in the same time period, avoiding excessive pressure on all nodes, and effectively reducing the impact on the production business of the distributed source database. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a data comparison method in an embodiment;

[0047] Figure 2 is a schematic diagram of a data migration process in an embodiment;

[0048] Figure 3 A schematic diagram of a flow chart of a step of determining a first sharding table in an embodiment;

[0049] Figure 4 is a structural block diagram of a data comparison device in an embodiment;

[0050] Figure 5 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In order to enable those skilled in the art to better understand the present application, the relevant technology is first introduced below.

[0053] With the development and update of database products and the changes in user needs, there will be a continuous demand for data migration. Users can use data migration tools to migrate data between different databases. Due to the complexity of the database system, data migration involves many details. In scenarios with high requirements for data accuracy, after migrating database objects and data from the source database to the target database, users often want to verify the accuracy of the migration work, which leads to the need to compare the data before and after the migration.

[0054] In the related art, the function of verifying the data before and after the migration provided by the data migration tool is basically for a one-to-one database topology, that is, the source database is a single database instance, and the target database is also a single database instance, for example, the data in a MySQL type database is migrated to another MySQL type database, or the data in a PostgreSQL type database is migrated to another PostgreSQL type database. For many-to-one database migration, such as migrating data from a distributed database to a PostgreSQL type database, the distributed database as the source database has multiple database instances (also called nodes of the distributed database). Faced with this topology, the data comparison method currently used by data migration tools is to simplify it into a one-to-one topology. The working process is: for the source database, it is considered as a single database instance as a whole, and the data of each database node underneath is directly extracted from the source database proxy layer, and compared with the data obtained by the target database migration.

[0055] This method is simple and intuitive, but it has some disadvantages. On the one hand, distributed databases often store a large amount of data. The above method needs to extract a large amount of data from the distributed database when performing data comparison. On the other hand, when extracting data from each database node, all database nodes in the distributed database consume resources for data comparison during the data comparison, which seriously hinders normal production business. It can be seen that the relevant technology has the problem of low data comparison efficiency.

[0056] Based on this, it is necessary to provide a data comparison method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0057] In one embodiment, Figure 1 As shown, a data comparison method is provided. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a system including a terminal and a server and implemented through the interaction between the terminal and the server. The server can be connected to the target database and the distributed source database through a network.

[0058] In this embodiment, the method includes the following steps:

[0059] S101, determining data to be compared.

[0060] The data to be compared are two sets of data to be compared. In this embodiment, the data to be compared may include data in the source database before data migration and data in the target database after data migration. In the data migration process, the data sending end is called the source database, and the data receiving end is called the target database. Data migration can be understood as copying the structure and data in a database from one database system to another database system, wherein structural migration is to migrate the object structure in the database, such as tables, views, functions, stored procedures and other objects. Data comparison can be understood as comparing whether the data in the source database and the target database involved in the data migration process are consistent.

[0061] In some embodiments, data migration may include one or more of full migration and incremental synchronization, wherein full migration refers to migrating offline data of a source database, and incremental synchronization refers to migrating real-time data of a source database.

[0062] S102, when the data to be compared includes a data table of the target database, determine the first shard table to be compared from the multiple shard tables contained in the data table; the data table is obtained by migrating data from the shard tables provided by multiple database nodes in the distributed source database, and the first shard table to be compared is a part of the multiple shard tables.

[0063] In practical applications, the data to be compared may include a shard table in a distributed source database and a data table in a target database. The data table in the target database may be obtained by combining shard tables provided by multiple database nodes in the distributed source database, for example, by combining information in shard tables provided by some or all database nodes. The data packet may also be referred to as a logical table.

[0064] The distributed source database may have one or more corresponding distributed database middleware. The distributed database middleware is a database middleware designed to solve the problem of performance degradation caused by excessive data volume in a single-machine database. The distributed database middleware can shard tables and store data in shards on multiple database instances (also called database nodes) at the bottom of the distributed database. Each database node may store one or more shard tables. In some embodiments, the distributed database middleware may have a corresponding distributed database middleware proxy layer, which is an application-oriented proxy program that is usually available in this type of distributed database middleware product, and can be used to manage metadata and data routing issues of sharded libraries and tables.

[0065] A shard table is a physical table stored in a database node in a distributed database. During subsequent data migration, the table information in the shard table becomes part of the logical table. In some examples, a physical table in a database node can be understood as a shard table, and all related physical table shards form a logical table. Therefore, the physical table used to form a logical table (or a database in the target database) is also called a shard table.

[0066] For example, see Figure 2 For a distributed database, the application sees the logical table name from the proxy layer, such as SchemaA.TableA. However, the table SchemaA.TableA does not exist on each database node under the distributed database. The actual situation may be that the logical table SchemaA.TableA is composed of physical tables SchemaA_1.TableA_1, SchemaA_1. TableA_2, SchemaA_2. TableA_1, and SchemaA_2. TableA_2. Among them, the physical tables SchemaA_1. TableA_1 and SchemaA_1. TableA_2 are in the schema SchemaA_1 of database node 1, and the physical tables SchemaA_2. TableA_1 and SchemaA_2. TableA_2 are in the schema SchemaA_2 of database node 2.

[0067] Of course, in actual applications, there may be other forms of data tables, such as single-chip tables and global tables. A single-chip table is a logical table composed of only one physical table shard, that is, the physical table that constitutes the logical table information exists on a single node of the distributed database, and the logical table name is the physical table name; the logical table name of the global table is the physical table name, and the same table structure and data exist on every node of the distributed database.

[0068] When it is determined that the data to be compared includes a data table of the target database, a first shard table to be compared can be determined from a plurality of shard tables included in the data table.

[0069] S103, obtaining a second shard table from a database node that provides the first shard table in the distributed source database, and comparing the second shard table with the first shard table.

[0070] Compared with the related art which does not care about or understand the topological structure of the internal details of the distributed database when comparing many-to-one data migration, and only considers the distributed source database as a whole as a single database instance, in this step, when it is determined that the data to be compared includes a data table composed of multiple shard tables, and the first shard table to be compared is determined, the database node that provides the first shard table can be determined in the distributed source database according to the correspondence between the shard tables in the data table and each database node, and then the corresponding shard table, i.e., the second shard table, is obtained from the database node, and then the first shard table and the second shard table are compared.

[0071] It should be noted that, with respect to the method in the related art of directly extracting data from each database node below the proxy layer and comparing it with the data extracted from the target database, the inventors found that this processing method would require the extraction of a large amount of data from the distributed source database and cause all database nodes to consume resources for data comparison, thereby seriously hindering normal production business. In this embodiment, by obtaining part of multiple shard tables in the data table, namely the first shard table, on the one hand, each of the multiple shard tables included can perform data comparison in different time periods, reducing the data comparison pressure on different database nodes; on the other hand, it can reduce the amount of data pulled from the distributed source database, thereby effectively avoiding obstruction to normal production business and improving data comparison efficiency.

[0072] In the above data comparison method, the server can determine the data to be compared. When the data to be compared includes the data table of the target database, the first shard table to be compared can be determined in the multiple shard tables contained in the data table, wherein the data table is obtained by performing data migration on the shard tables provided by multiple database nodes in the distributed source database, and the first shard table to be compared is part of the multiple shard tables; then the second shard table is obtained in the database node that provides the first shard table in the distributed source database, and the second shard table is compared with the first shard table. In the embodiment of the present application, when the data to be compared includes the data table of the target database, after determining part of the shard tables in the data table as the first shard table to be compared, the second shard table is pulled from the corresponding database node for comparison, and data comparison can be performed for each shard data of the distributed database node in different time periods according to the corresponding relationship between the shard table and the database node, so as to avoid the content of the data comparison task involving the full set of data of all database nodes in the same time period, avoid excessive pressure on all nodes, and effectively reduce the impact on the production business of the distributed source database.

[0073] In one embodiment, Figure 3 As shown, in step S102, determining the first shard table to be compared among the multiple shard tables included in the data table may include the following steps:

[0074] S301, determining a target database node for data comparison among multiple database nodes of a distributed source database, and determining shard table source information according to the target database node; the target database node is part of the multiple database nodes.

[0075] In a specific implementation, when comparing a distributed source database with a target database obtained after migration, a target database node for data comparison can be determined from among multiple database nodes of the distributed source database, wherein there are one or more target database nodes, and the target database nodes can be manually selected by a user, or some nodes can be screened out by the server according to the node selection logic.

[0076] After determining the target database node, it can be determined that the subsequent step is to compare the data provided by the target database node and the corresponding data in the data table. Therefore, the shard table source information can be determined based on the target database node, wherein the shard table source information can be information that characterizes the source of the shard table data. For example, if the target database node is node A, the shard table source information can be information indicating that the shard table comes from node A.

[0077] S302, filtering multiple shard tables included in the data table according to the shard table source information, and obtaining a shard table from the target database node as the first shard table.

[0078] Furthermore, the shard table source information can be used to filter multiple shard tables included in the data table, and the shard table from the target database node obtained by filtering is used as the first shard table.

[0079] In this embodiment, by taking some of the multiple database nodes as target database nodes and then filtering out the first shard table based on the target database nodes, it is possible to avoid pulling data and comparing data for all database nodes of the distributed source database at the same time, effectively reducing the computing pressure of the database nodes, avoiding consuming a large amount of resources for data comparison, and helping to ensure that other businesses of the distributed source database can proceed normally.

[0080] In one embodiment, in step S301, determining a target database node to be compared with data among multiple database nodes of a distributed source database may include the following steps:

[0081] In response to a database node selection instruction triggered by a user, a database node selected by the user is determined as a target database node among multiple database nodes of a distributed source database; or, data pulling periods of each of the multiple database nodes of the distributed source database are determined; the multiple data pulling periods include different data pulling periods; for each data pulling period, when the current time matches the data pulling period, the database node associated with the data pulling period is used as the target database node.

[0082] In practical applications, a database node selection instruction for selecting a database node can be triggered by a user, and the server responds to the instruction by determining the database node selected by the user as the target database node among multiple database nodes of the distributed source database. In some embodiments, the database node selected by the user does not exceed the node quantity threshold, and the node quantity threshold can be determined according to the data processing capability of the distributed source database. For example, the node quantity threshold can be determined according to the total amount of idle computing resources of the distributed source database. The node quantity threshold is positively correlated with the total amount of idle computing resources, so that when the distributed source database has more idle computing resources, more database nodes can be allowed to participate in the data comparison process. When the distributed source database has fewer idle computing resources, the database nodes participating in the data comparison are reduced, which can ensure that enough database nodes are retained to participate in normal production business.

[0083] In some other embodiments, a data pulling period configuration of a database node may be provided to the server in advance, and the configuration may record the data pulling periods of each of the multiple database nodes in the distributed source database, wherein the multiple data pulling periods may include different data pulling periods, that is, in the multiple data pulling periods, some or all of the data pulling periods are different. Then, for each data pulling period, when the current time matches the data pulling period, the database node associated with the data pulling period may be used as the target database node.

[0084] In this embodiment, the user can flexibly select some database nodes as target database nodes, or some database nodes can be used as target database nodes according to the data pulling period, which effectively avoids all database nodes of the distributed source database from participating in the data comparison and ensures the normal operation of other services of the distributed source database.

[0085] In one embodiment, the data table includes a data source field, such as Figure 2 As shown, the data table in the target database may include a data source field "TAG", and the field information under this field may represent the data source of the shard table, that is, the database node that provides the shard table.

[0086] Accordingly, in step S302, filtering multiple shard tables included in the data table according to the shard table source information to obtain the shard table from the target database node may include the following steps:

[0087] For each shard table contained in the data table, determine the field information of the shard table under the data source field; match the field information of each shard table with the shard table source information, and obtain the shard table from the target database node according to the shard table corresponding to the successfully matched field information.

[0088] In actual applications, when migrating data from a distributed source database to a target database and generating a corresponding data table, a data source field can be set in the data table to record the source of each shard table, and then to determine the source information of the shard table. For each shard table contained in the data table, the field information of the shard table under the data source field can be read, and then the field information of each shard table can be matched with the shard table source information, and the shard table corresponding to the successfully matched field information can be determined as the shard table from the target database node.

[0089] In this embodiment, through the field information under the data source field in the data table, the shard table from the target database node can be quickly screened out from the data table, while effectively reducing the number of shard table comparisons, the shard table related to the current data comparison can be accurately determined.

[0090] In one embodiment, before step S101, the method may further include the following steps:

[0091] During the data migration phase, the data source identifier corresponding to each database node is determined; based on the sharding tables provided by multiple database nodes, a data table containing data source fields is generated, and the data source identifier of each database node providing the sharding table is used as the field information of the data source field of the corresponding sharding table.

[0092] In the specific implementation, during the data migration stage, the data source identifier corresponding to each database node can be determined in advance, where the data source identifier is used to indicate that the data source is the corresponding database node. The data source identifier can be unique, so that the data source identifiers of different database nodes can be distinguished, so that the database node that provides the shard table can be indicated later through the data source identifier.

[0093] Then, when the logical table is determined based on the shard tables provided by multiple database nodes and migrated to the target database to obtain the corresponding data table, a data source field can be added to the data table, and the data source identifier of each database node that provides the shard table can be added to the data source field of the shard table corresponding to the data table as the field information of the data source field.

[0094] In this embodiment, during the data migration stage, an additional field is added to the table object migrated to the target database to record the source node of each data, and the data source identifier is used as the field information under the data source field of the corresponding shard table. Through this field, part of the data corresponding to a shard of the source end migrated to this database can be defined on the target database, thereby realizing shard comparison of the shard table data, and providing a basis for quickly determining specific data shards and related database nodes during subsequent data comparison.

[0095] In one embodiment, determining the data source identifier corresponding to each database node may include the following steps:

[0096] Determine the IP address and port information corresponding to each database node; for each database node, obtain the data source identifier corresponding to the database node based on the splicing result of the IP address and port information corresponding to the database node.

[0097] In the specific implementation, for each database node, the IP address and port information corresponding to the database node can be obtained, and then the data source identifier corresponding to the database node can be obtained based on the splicing result of the IP address and port information. Thus, while quickly obtaining the data source identifier, the correspondence between the data source identifier and the database node can be effectively ensured, so as to facilitate the distinction and screening of shard tables through the data source identifier.

[0098] In order to enable those skilled in the art to better understand the above steps, the embodiment of the present application is illustrated below by using an example, but it should be understood that the embodiment of the present application is not limited to this.

[0099] refer to Figure 2 In actual applications, users use the data migration system to migrate data. Before data migration, they can configure the information of several steps involved in the database migration task on the system console, which involves the configuration items of the shard comparison data after the distributed database middleware data migration. The following are the steps for configuring the task, including:

[0100] (1) The user fills in the connection information of the source database and the target database. The data migration system can identify whether the source database is a distributed database and the specific node topology structure under the distributed database.

[0101] (2) The user selects the migration phase required for this migration task: structural migration, full migration, or incremental synchronization. The user can select any combination of structural migration, full migration, and incremental synchronization. In one example, if the user does not select the structural migration phase, the task does not support sharding data comparison for the distributed database, and the traditional full data comparison method will be used to compare the data on all nodes.

[0102] (3) The user selects the source database object that he wants to migrate to the target database. The data migration system can identify whether the selected source database table object is a sharded table, a single-shard table, or a global table.

[0103] (4) The data migration system can provide the following configuration for each selected source database shard table: During the structural migration phase, when migrating the table structure of the source shard table to the target end, whether to add a table field (i.e., data source field) based on the table structure of the source database to record the node information of the source distributed database from which the data row of the target database comes. In some examples, in order to avoid conflicts with the field name of the table itself, the field name actually added can have a prefix unique to the database migration system. Figure 2 The data source field "TAG" in the example does not have this prefix added. At this point, the task configuration is complete.

[0104] After the task configuration is completed, the user can start the task. The following are the relevant steps in the task running phase, including:

[0105] (1) Structural migration phase: When migrating the table structure of the source shard table to the target end, the migration thread adds a table field named TAG based on the table structure of the source database according to the task configuration information.

[0106] (2) Full migration stage: This stage is an optional stage when configuring the task. If full migration is configured, the data migration worker thread pulls the existing data from the source database and adds the insertion value of the TAG field that the data migration system creates for the target table structure when generating INSERT SQL based on the original data. The value of the TAG field is the source database source node information of the data, which can be, for example, information obtained by combining "IP+PORT information" or other database node identification information.

[0107] (3) Incremental synchronization phase: This phase is optional when configuring tasks. If incremental synchronization is configured, the data migration worker thread pulls transaction logs from the source database and parses the original semantics. If it is an INSERT type DML log event, it generates INSERT SQL based on the original value and adds the insertion value of the TAG field that the data migration system creates for the target table structure.

[0108] When the task runs to the incremental synchronization stage, or when there is no incremental synchronization stage and the full migration stage has been completed, you can create a data comparison task as needed.

[0109] During the data comparison task, users can create a data comparison task and select the source table object and its corresponding target table object to be compared. For the table object to be compared, based on whether the source table object is a sharded table and whether the corresponding target table object has an additional TAG field created, users can be prompted to choose whether to use the traditional full data comparison method or the sharded data comparison method. If the user chooses to use the sharded data comparison method, the user will be prompted to further select which node of the source distributed database the sharded data is to be compared.

[0110] According to the parameters selected by the user, if the shard data comparison method is used, for the shard table, the data comparison thread pulls the shard data of the table object involved on the node of the source distributed database specified by the user, and at the same time, when pulling data on the corresponding table object of the target database, the SQL filter condition will be used to filter the TAG field to the data from the source node, thereby ensuring that the data range of the two sides is consistent. For the data pulled from the source and target ends, it can be sorted by the primary key or unique key of the table, thereby ensuring that the data order of the two sides is one-to-one. After determining the data range and data order, the two parts of the data can be compared and the comparison results can be recorded.

[0111] Compared with the traditional method of comparing data before and after the migration of the distributed database middleware, the embodiment of the present application can perform data comparison on each shard data of the distributed database middleware in different time periods, so as to avoid the data comparison task in the same time period involving the entire data set of all database nodes, causing excessive pressure on all nodes, and reducing the impact on production business.

[0112] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0113] Based on the same inventive concept, the embodiment of the present application also provides a data comparison device for implementing the data comparison method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data comparison device embodiments provided below can refer to the limitations on the data comparison method above, and will not be repeated here.

[0114] In an exemplary embodiment, Figure 4 As shown, a data comparison device is provided, comprising:

[0115] The data acquisition module 401 is used to determine the data to be compared;

[0116] The shard acquisition module 402 is used to determine a first shard table to be compared from among multiple shard tables contained in the data table when the data to be compared includes a data table of a target database; the data table is obtained by performing data migration on shard tables provided by multiple database nodes in a distributed source database, and the first shard table to be compared is part of the multiple shard tables;

[0117] The shard comparison module 403 is used to obtain a second shard table from the database node that provides the first shard table in the distributed source database, and compare the second shard table with the first shard table.

[0118] In one embodiment, the fragment acquisition module 402 is used to:

[0119] Determine a target database node to be compared with data among the multiple database nodes of the distributed source database, and determine the shard table source information according to the target database node; the target database node is part of the multiple database nodes;

[0120] According to the shard table source information, multiple shard tables included in the data table are filtered to obtain a shard table from the target database node as the first shard table.

[0121] In one embodiment, the fragment acquisition module 402 is used to:

[0122] In response to a database node selection instruction triggered by a user, determining a database node selected by the user as a target database node among the multiple database nodes of the distributed source database;

[0123] or,

[0124] Determine a data pulling period for each of the plurality of database nodes of the distributed source database; the plurality of data pulling periods include different data pulling periods;

[0125] For each of the data pulling periods, when the current time matches the data pulling period, the database node associated with the data pulling period is used as the target database node.

[0126] In one embodiment, the data table includes a data source field;

[0127] The fragment acquisition module 402 is used to:

[0128] For each of the shard tables included in the data table, determining field information of the shard table under the data source field;

[0129] The field information of each shard table is matched with the shard table source information, and the shard table from the target database node is obtained according to the shard table corresponding to the successfully matched field information.

[0130] In one embodiment, the device further includes a field information adding module, wherein the field information adding module is used to:

[0131] In the data migration phase, a data source identifier corresponding to each database node is determined; the data source identifier is used to indicate that the data source is the corresponding database node;

[0132] A data table including a data source field is generated according to the shard tables provided by the plurality of database nodes, and the data source identifier of each database node providing the shard table is used as field information of the data source field of the corresponding shard table.

[0133] In one embodiment, the field information adding module is used to:

[0134] Determine the IP address and port information corresponding to each of the database nodes;

[0135] For each of the database nodes, a data source identifier corresponding to the database node is obtained based on the splicing result of the IP address and port information corresponding to the database node.

[0136] Each module in the above data comparison device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0137] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the target database. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data comparison method is implemented.

[0138] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0141] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0143] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0145] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data comparison method, characterized in that: The method comprises: Determine the data to be compared; When the data to be compared includes a data table of a target database, a first shard table to be compared is determined from among the multiple shard tables included in the data table; the data table is obtained by performing data migration on shard tables provided by multiple database nodes in a distributed source database, and the first shard table to be compared is part of the multiple shard tables; A second shard table is obtained from a database node of the distributed source database that provides the first shard table, and the second shard table is compared with the first shard table.

2. The method according to claim 1, characterized in that The step of determining a first shard table to be compared among the multiple shard tables included in the data table includes: Determine a target database node to be compared with data among the multiple database nodes of the distributed source database, and determine the shard table source information according to the target database node; the target database node is part of the multiple database nodes; According to the shard table source information, multiple shard tables included in the data table are filtered to obtain a shard table from the target database node as the first shard table.

3. The method according to claim 2, characterized in that The step of determining a target database node to be compared with data from among the plurality of database nodes of the distributed source database includes: In response to a database node selection instruction triggered by a user, determining a database node selected by the user as a target database node among the multiple database nodes of the distributed source database; or, Determine a data pulling period for each of the plurality of database nodes of the distributed source database; the plurality of data pulling periods include different data pulling periods; For each of the data pulling periods, when the current time matches the data pulling period, the database node associated with the data pulling period is used as the target database node.

4. The method according to claim 2, characterized in that: The data table includes a data source field; The filtering of the multiple shard tables contained in the data table according to the shard table source information to obtain the shard table from the target database node includes: For each of the shard tables included in the data table, determining field information of the shard table under the data source field; The field information of each shard table is matched with the shard table source information, and the shard table from the target database node is obtained according to the shard table corresponding to the successfully matched field information.

5. The method according to claim 4, characterized in that Before the step of determining the data to be compared, the method further includes: In the data migration phase, a data source identifier corresponding to each database node is determined; the data source identifier is used to indicate that the data source is the corresponding database node; A data table including a data source field is generated according to the shard tables provided by the plurality of database nodes, and the data source identifier of each database node providing the shard table is used as field information of the data source field of the corresponding shard table.

6. The method according to claim 5, characterized in that The determining of the data source identifier corresponding to each database node includes: Determine the IP address and port information corresponding to each of the database nodes; For each of the database nodes, a data source identifier corresponding to the database node is obtained based on the splicing result of the IP address and port information corresponding to the database node.

7. A data comparison device, characterized in that: The device comprises: A data acquisition module, used to determine the data to be compared; A shard acquisition module is used to determine a first shard table to be compared from among multiple shard tables contained in the data table when the data to be compared includes a data table of a target database; the data table is obtained by performing data migration on shard tables provided by multiple database nodes in a distributed source database, and the first shard table to be compared is part of the multiple shard tables; The shard comparison module is used to obtain a second shard table from the database node that provides the first shard table in the distributed source database, and compare the second shard table with the first shard table.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images