Heterogeneous database data processing methods, apparatus, and computer equipment

Abstract

This application relates to a method, apparatus, and computer device for processing heterogeneous database data, and pertains to the fields of fintech or other related fields. The method includes: obtaining comparison range information for the current data comparison task; determining comparison data table queues corresponding to multiple comparison rule information based on the type classification results of each first data table in a first data table set; processing the multiple comparison data table queues through multi-threading to obtain the rule execution results corresponding to each comparison data table queue; obtaining the heterogeneous database comparison result corresponding to the current data comparison task based on the first and second execution results of each rule execution result; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database. This method achieves rapid comparison of data integrity in heterogeneous databases, enabling efficient and reliable comparison of massive amounts of data, and improving data migration efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, storage medium, and computer program product for processing heterogeneous database data. Background Technology

[0002] Oracle databases were among the first commercial databases used in financial institutions, capable of supporting massive amounts of data and complex business processes. However, Oracle databases have limitations and security issues, necessitating replacement with other databases, such as Gaussian Database.

[0003] In related technologies, the migration of massive amounts of data between heterogeneous databases presents challenges such as long time consumption for integrity comparison and high manpower investment. Furthermore, the low data processing efficiency of heterogeneous databases cannot effectively guarantee the integrity of data migration. Summary of the Invention

[0004] Therefore, it is necessary to provide a heterogeneous database data processing method, apparatus, computer equipment, storage medium, and computer program product that can solve the above-mentioned technical problems.

[0005] Firstly, this application provides a method for processing heterogeneous database data, the method comprising:

[0006] Obtain the comparison range information for the current data comparison task; the comparison range information is used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, wherein the data tables in the first set of data tables correspond one-to-one with the data tables in the second set of data tables, and the first set of data tables is determined based on the data migration situation of the data source database;

[0007] Based on the type classification results corresponding to each first data table in the first data table set, a comparison data table queue corresponding to each of the multiple comparison rule information is determined; each comparison data table queue includes at least one first data table and at least one second data table corresponding to the at least one first data table;

[0008] Multiple comparison data table queues are processed by multiple threads to obtain the rule execution result corresponding to each comparison data table queue; the rule execution result includes a first execution result based on at least one first data table and a second execution result based on at least one second data table;

[0009] Based on the first execution result and the second execution result in the execution results of each rule, the heterogeneous database comparison result corresponding to the current data comparison task is obtained; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database.

[0010] In one embodiment, obtaining the comparison range information of the current data comparison task includes:

[0011] Obtain migration plan information and actual migration information of the data source database; the migration plan information is used to characterize the data tables to be migrated in the data source database, and the actual migration information is used to characterize the data tables that have been migrated in the data source database at the start of the current data comparison task;

[0012] Based on the migration plan information and the actual migration information, a first set of data tables is determined from the data source database to obtain the comparison range information for the current data comparison task.

[0013] In one embodiment, determining the comparison data table queues corresponding to each of the multiple comparison rule information based on the type classification results corresponding to each first data table in the first data table set includes:

[0014] Based on the data change information corresponding to each of the first data tables, the type classification results corresponding to each of the first data tables are obtained; different type classification results correspond to different comparison rule information;

[0015] Based on the type classification results corresponding to each of the first data tables, multiple comparison rule information is determined;

[0016] By combining the multiple comparison rule information, the first data table set, and the second data table set, the multiple comparison data table queues are determined.

[0017] In one embodiment, determining the multiple comparison data table queues by combining the multiple comparison rule information, the first data table set, and the second data table set includes:

[0018] For each comparison rule, at least one first data table is determined from the first data table set, and at least one second data table corresponding to the at least one first data table is determined from the second data table set;

[0019] Based on the at least one first data table and the at least one second data table, a comparison data table queue corresponding to the comparison rule information is obtained.

[0020] In one embodiment, obtaining the rule execution results corresponding to each of the comparison data table queues includes:

[0021] When the comparison data table queue includes a first data table and a second data table, the first data table and the second data table are compared according to the first comparison rule to obtain the rule execution result corresponding to the comparison data table queue.

[0022] In one embodiment, obtaining the rule execution results corresponding to each of the comparison data table queues includes:

[0023] When the comparison data table queue includes at least two first data tables and at least two second data tables, the first processed data table corresponding to the at least two first data tables and the second processed data table corresponding to the at least two second data tables are obtained according to the second comparison rule.

[0024] By comparing the first processed data table and the second processed data table, the rule execution results corresponding to the comparison data table queue are obtained.

[0025] In one embodiment, the heterogeneous database comparison result includes multiple data table comparison results. The step of obtaining the heterogeneous database comparison result corresponding to the current data comparison task based on the first execution result and the second execution result among the execution results of each rule includes:

[0026] For each rule execution result, the first execution result and the second execution result are analyzed to obtain the first feature value corresponding to the first execution result and the second feature value corresponding to the second execution result;

[0027] Based on the comparison results of the first feature value and the second feature value, the comparison results of the data table corresponding to the comparison data table queue to which the rule execution result belongs are obtained.

[0028] Secondly, this application also provides a heterogeneous database data processing apparatus, the apparatus comprising:

[0029] The comparison range acquisition module is used to acquire the comparison range information of the current data comparison task; the comparison range information is used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, wherein the data tables in the first set of data tables correspond one-to-one with the data tables in the second set of data tables, and the first set of data tables is determined according to the data migration situation of the data source database;

[0030] The comparison rule determination module is used to determine the comparison data table queues corresponding to each of the multiple comparison rule information based on the type classification results corresponding to each first data table in the first data table set; each comparison data table queue includes at least one first data table and at least one second data table corresponding to the at least one first data table;

[0031] The rule execution result obtaining module is used to process multiple comparison data table queues through multi-threading to obtain the rule execution result corresponding to each comparison data table queue; the rule execution result includes a first execution result obtained based on at least one first data table and a second execution result obtained based on at least one second data table;

[0032] The comparison result obtaining module is used to obtain the heterogeneous database comparison result corresponding to the current data comparison task based on the first execution result and the second execution result in the execution results of each rule; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database.

[0033] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the heterogeneous database data processing method described above.

[0034] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the heterogeneous database data processing method described above.

[0035] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the heterogeneous database data processing method described above.

[0036] The aforementioned heterogeneous database data processing method, apparatus, computer equipment, storage medium, and computer program product acquire comparison range information for the current data comparison task. This comparison range information indicates a first set of data tables in the data source database to be compared and a second set of data tables in the data import database. The data tables in the first set correspond one-to-one with the data tables in the second set. The first set of data tables is determined based on the data migration status of the data source database. Then, based on the type classification results corresponding to each first data table in the first set, multiple comparison rule information corresponding to different comparison data table queues are determined. Each comparison data table queue includes at least one first data table and at least one second data table corresponding to at least one first data table. By processing multiple comparison data table queues through multi-threading, the system obtains the rule execution results corresponding to each comparison data table queue. These rule execution results include a first execution result based on at least one first data table and a second execution result based on at least one second data table. Then, based on the first and second execution results in each rule execution result, the system obtains the heterogeneous database comparison result corresponding to the current data comparison task. This heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database, realizing rapid comparison of data integrity in heterogeneous databases. The system automatically compares and obtains rule execution results based on the comparison data table queues corresponding to the comparison rule information, enabling efficient and reliable comparison of massive amounts of data and improving data migration efficiency. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating a heterogeneous database data processing method in one embodiment;

[0038] Figure 2 This is a schematic diagram of a heterogeneous database comparison process in one embodiment;

[0039] Figure 3a This is a schematic diagram of an execution comparison rule process in one embodiment;

[0040] Figure 3b This is a schematic diagram of a process for parsing and comparing results in one embodiment;

[0041] Figure 4 This is a flowchart illustrating another heterogeneous database data processing method in one embodiment;

[0042] Figure 5 This is a structural block diagram of a heterogeneous database data processing device in one embodiment;

[0043] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties; correspondingly, this application also provides a corresponding user authorization entry point for users to choose to authorize or refuse.

[0046] In one embodiment, such as Figure 1 As shown, a method for processing heterogeneous database data is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0047] Step 101: Obtain the comparison range information for the current data comparison task;

[0048] The comparison range information can be used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, with each data table in the first set of data tables corresponding one-to-one with the data tables in the second set of data tables.

[0049] As an example, the first set of data tables can be determined based on the data migration status of the data source database. For example, the first set of data tables can be determined from the data source database by obtaining the migration plan information and actual migration information of the data source database.

[0050] In practical applications, by obtaining the migration plan information of the data table to be migrated in the data source database, as well as the actual migration information of the data table that has been migrated in the data source database at the start of the current data comparison task, the data table to be compared can be determined from multiple data tables in the data source database as the first set of data tables corresponding to the data source database. Then, the comparison range information of the current data comparison task can be obtained based on the first set of data tables.

[0051] Specifically, the data source database to be compared can be an Oracle database, and the data import database can be a Gaussian database, such as... Figure 2As shown, the comparison range information can be determined for the current data comparison task. For example, the DBA (Database Administrator) can select the data tables to be compared to set the comparison range, or the set comparison range can be automatically obtained in response to the triggering of the current data comparison task.

[0052] In one example, based on the Oracle database table migration plan and the actual table migration status, tables that are not planned for migration can be excluded from the comparison. The table migration plan indicates the data migration work that needs to be completed before a specified time, while the actual table migration status indicates the tables that have been migrated before the data comparison is performed.

[0053] In an optional embodiment, the data tables that have been migrated and the data tables that are planned to be migrated can be distinguished again to exclude the transit tables that are used to temporarily store data from the comparison scope. The transit tables are used to store intermediate data for calculation. The data stored in them cannot be used as final data, but are used as intermediate products of data processing, and can be cleaned up periodically.

[0054] Step 102: Based on the type classification results of each first data table in the first data table set, determine the comparison data table queues corresponding to each of the multiple comparison rule information.

[0055] Each comparison data table queue may include at least one first data table and at least one second data table corresponding to the at least one first data table.

[0056] In the specific implementation, the type classification result corresponding to each first data table can be determined based on the data change information corresponding to each first data table in the first data table set. Since different type classification results correspond to different comparison rule information, multiple comparison rule information can be determined based on the type classification results corresponding to each first data table. Then, by combining multiple comparison rule information and the one-to-one correspondence between the data tables in the first data table set and the second data table set, the comparison data table queues corresponding to each of the multiple comparison rule information can be obtained.

[0057] For example, such as Figure 2 As shown, comparison rules can be set by classifying the Oracle database tables within the comparison range (i.e., according to the classification results of the types of each first data table in the first data table set). Different data comparison rules (i.e., comparison rule information) can be set for different types of data tables based on business rules and data migration execution.

[0058] In one example, the comparison rule information may include a first comparison rule for simple data comparison, which can perform a simple one-to-one comparison of the data tables to be compared in the data source database and the data import database; the comparison rule information may also include a second comparison rule for complex business data comparison, which can logically associate the data of N tables according to business logic to form a fixed comparison data column, and then perform an N-to-N logical comparison of the data tables to be compared in the data source database and the data import database. The comparison rule information may also include other comparison rules, which are not specifically limited in this embodiment.

[0059] Step 103: Process multiple comparison data table queues using multi-threading to obtain the rule execution results corresponding to each comparison data table queue;

[0060] As an example, the rule execution result may include a first execution result obtained based on at least one first data table and a second execution result obtained based on at least one second data table.

[0061] After obtaining the comparison data table queue, multiple comparison data table queues can be processed using multi-threading. For example, each comparison thread can execute one comparison rule, with each thread execution set as the smallest unit. If the comparison data table queue includes a first data table and a second data table, the first data table and the second data table can be compared according to the first comparison rule to obtain the rule execution result corresponding to the comparison data table queue. Alternatively, if the comparison data table queue includes at least two first data tables and at least two second data tables, the first processed data table corresponding to at least two first data tables and the second processed data table corresponding to at least two second data tables can be obtained according to the second comparison rule. Then, the first processed data table and the second processed data table can be compared to obtain the rule execution result corresponding to the comparison data table queue.

[0062] In one example, such as Figure 2 As shown, comparison rules can be executed within a range. By using multi-threading technology to connect to the database based on the data source database and the data import database, comparison rules can be executed in parallel. Data to be compared can be extracted on a rule-by-rule basis, and the execution result of each comparison rule can be saved as an independent execution result file (i.e., rule execution result). Each execution result file can be stored separately according to the rule, source, and execution time.

[0063] Step 104: Based on the first execution result and the second execution result in each of the rule execution results, obtain the heterogeneous database comparison result corresponding to the current data comparison task.

[0064] Among them, the results of the heterogeneous database comparison can be used to characterize the differences in data migration between the data source database and the data import database.

[0065] In practical applications, for each rule execution result, the first and second execution results can be parsed to obtain the first feature value corresponding to the first execution result and the second feature value corresponding to the second execution result. Then, based on the comparison result of the first and second feature values, the data table comparison result corresponding to the comparison data table queue to which the rule execution result belongs can be obtained. Furthermore, based on the comparison results of each data table, the heterogeneous database comparison result corresponding to the current data comparison task can be obtained.

[0066] Specifically, such as Figure 2 As shown, the comparison execution result file (i.e., the rule execution result) can be SHA-2 encoded to parse the comparison execution result. By comparing the SHA-2 encoded value of the comparison execution result file of the data source database with the SHA-2 encoded value of the comparison execution result file of the corresponding imported data in the imported database, if the SHA-2 encoded values ​​are the same, it can be confirmed that the source data and imported data of the compared data table are completely identical; if the SHA-2 encoded values ​​are different, it can be confirmed that the source data and imported data of the compared data table are not completely identical. This enables rapid comparison of data integrity between Gaussian database and Oracle database, solving the problem of time-consuming and labor-intensive integrity comparison of massive amounts of data between heterogeneous databases, and providing a guarantee for the migration of massive amounts of data in heterogeneous databases.

[0067] Compared to traditional methods, the technical solution in this embodiment, by setting data comparison rules to execute and parse the comparison results, can automatically compare data in the data source database and data imported into the database, and identify the differences and similarities in the data. This allows for automated verification to replace traditional manual comparison verification, reducing the waiting time in the process flow. The average processing time for a single table is approximately 2 hours, compared to the traditional average verification and comparison time of 1-2 days, achieving a reduction in verification and comparison time from the days to the hours. Furthermore, by fully utilizing automation, it reduces manpower input and improves the accuracy of data verification and comparison.

[0068] In the above-described heterogeneous database data processing method, the comparison range information of the current data comparison task is obtained. Then, based on the type classification results of each first data table in the first data table set, the comparison data table queues corresponding to multiple comparison rule information are determined. Multiple comparison data table queues are processed by multi-threading to obtain the rule execution results corresponding to each comparison data table queue. Then, based on the first and second execution results of each rule execution result, the heterogeneous database comparison result corresponding to the current data comparison task is obtained. This achieves rapid comparison of data integrity of heterogeneous databases. Automatic comparison of rule execution results is performed based on the comparison data table queues corresponding to the comparison rule information. This method can efficiently and reliably compare massive amounts of data, improving data migration efficiency.

[0069] In one embodiment, obtaining the comparison range information for the current data comparison task may include the following steps:

[0070] Obtain migration plan information and actual migration information from the data source database; based on the migration plan information and the actual migration information, determine a first set of data tables from the data source database to obtain the comparison range information for the current data comparison task.

[0071] Among them, migration plan information can be used to characterize the data tables to be migrated in the data source database; actual migration information can be used to characterize the data tables that have been migrated in the data source database at the start of the current data comparison task.

[0072] In practical applications, the DBA selects the data tables to be compared to set the comparison range. Based on the Oracle database's data table migration plan (i.e., migration plan information) and the actual data table migration status (i.e., actual migration information), data tables that are not planned to be migrated are excluded from the comparison range. Furthermore, by further distinguishing between data tables that have been migrated and data tables that are planned to be migrated, intermediate tables used as temporary data storage are excluded from the comparison range. Thus, the comparison range information for the current data comparison task can be determined.

[0073] In this embodiment, by obtaining migration plan information and actual migration information from the data source database, and then determining the first data table set from the data source database based on the migration plan information and actual migration information, the comparison range information of the current data comparison task can be obtained, which can accurately obtain the comparison range and provide data support for subsequent data comparison.

[0074] In one embodiment, determining the comparison data table queues corresponding to each of the multiple comparison rule information based on the type classification results of each first data table in the first data table set may include the following steps:

[0075] Based on the data change information corresponding to each of the first data tables, the type classification results corresponding to each of the first data tables are obtained; based on the type classification results corresponding to each of the first data tables, multiple comparison rule information is determined; combining the multiple comparison rule information, the set of first data tables, and the set of second data tables, the multiple comparison data table queues are determined.

[0076] Different classification results correspond to different comparison rules.

[0077] In one example, based on the data table classification rules, the DBA can set an inspection date and classify the data tables that have not undergone data changes before the inspection date (i.e., according to the data change information corresponding to each first data table) into cold data tables (i.e., type classification results) according to the Oracle log. This type of data table can be a dictionary table, which is suitable for using simple data comparison rules (i.e., comparison rule information), such as the first comparison rule for simple data comparison.

[0078] In another example, data tables that have undergone significant data changes before and after the inspection date can be classified as "hot data tables" based on Oracle logs. If multiple data tables change data simultaneously or before and after the date, they can be classified as hot data tables. This type of data can be business data tables, which are suitable for using complex business data comparison rules (i.e., comparison rule information). This is the second comparison rule for comparing complex business data.

[0079] By classifying the Oracle database tables within the comparison range to set comparison rules, different data comparison rules (i.e., comparison rule information) can be set for different types of tables based on business rules and data migration execution. Then, the comparison rules can be executed within the range. By extracting the data to be compared in units of comparison rules, the comparison data table queue corresponding to each comparison rule information can be obtained.

[0080] In this embodiment, by obtaining the type classification results corresponding to each first data table based on the data change information corresponding to each first data table, and then determining multiple comparison rule information based on the type classification results corresponding to each first data table, and then combining the multiple comparison rule information, the first data table set, and the second data table set, multiple comparison data table queues are determined. The comparison data table queues can be extracted based on the comparison rules, providing data support for subsequent data comparison.

[0081] In one embodiment, determining the multiple comparison data table queues by combining the multiple comparison rule information, the first data table set, and the second data table set may include the following steps:

[0082] For each comparison rule information, at least one first data table is determined from the first data table set, and at least one second data table corresponding to the at least one first data table is determined from the second data table set; based on the at least one first data table and the at least one second data table, a comparison data table queue corresponding to the comparison rule information is obtained.

[0083] In one example, a simple data comparison rule can be a case where a single table can be compared without any calculations or processing. For a simple data comparison rule, a first data table can be determined from a first data table set, and a second data table corresponding to the first data table can be determined from a second data table set, thus obtaining the comparison data table queue corresponding to the simple data comparison rule.

[0084] In another example, a complex business data comparison rule can be a situation where data needs to be extracted from multiple tables according to specific rules and then compared after being processed according to specified rules (such as deadline, data origin, etc.). For a complex business data comparison rule, multiple first data tables can be determined from a first data table set, and multiple second data tables corresponding to these multiple first data tables can be determined from a second data table set, thus obtaining a comparison data table queue corresponding to the complex business data comparison rule.

[0085] In this embodiment, for each comparison rule information, at least one first data table is determined from the first data table set, and at least one second data table corresponding to the at least one first data table is determined from the second data table set. Then, based on the at least one first data table and the at least one second data table, a comparison data table queue corresponding to the comparison rule information is obtained. This can automatically extract the corresponding comparison data table queue based on different data comparison rules, which helps to flexibly compare data changes.

[0086] In one embodiment, obtaining the rule execution results corresponding to each of the comparison data table queues may include the following steps:

[0087] When the comparison data table queue includes a first data table and a second data table, the first data table and the second data table are compared according to the first comparison rule to obtain the rule execution result corresponding to the comparison data table queue.

[0088] In practical applications, such as Figure 3aAs shown, for the simple data comparison rule (i.e. the first comparison rule), the data columns can be trimmed according to the actual data migration situation to perform a simple one-to-one comparison between the data source database and the data import database. For example, Table 1 can be sorted in descending order according to the time of data generation, and then it can be compared one-to-one with the migrated Table N in descending order of data generation.

[0089] In this embodiment, when the comparison data table queue includes a first data table and a second data table, the first data table and the second data table are compared according to the first comparison rule to obtain the rule execution result corresponding to the comparison data table queue. This can effectively compare simple data and improve data migration efficiency.

[0090] In one embodiment, obtaining the rule execution results corresponding to each of the comparison data table queues may include the following steps:

[0091] In the case where the comparison data table queue includes at least two first data tables and at least two second data tables, according to the second comparison rule, the first processed data table corresponding to the at least two first data tables and the second processed data table corresponding to the at least two second data tables are obtained; the first processed data table and the second processed data table are compared to obtain the rule execution result corresponding to the comparison data table queue.

[0092] In practical applications, for complex business data comparison rules (i.e., the second comparison rule), the data of N tables can be logically associated according to 0 business logic to form a fixed comparison data column, and then an N-to-N logical comparison can be performed on the data source database and the data import database to be compared data tables.

[0093] For example, data from multiple tables (A, B, C, etc., i.e., at least two primary tables) can be extracted based on primary keys, foreign keys, and business relationships. Then, data can be processed according to...

[0094] The average squared difference or summation is performed on the data from the regions where the data originated to obtain the first processed data table. Then, the same processing can be performed on the data in the five migrated data tables (i.e., at least two second data tables) for comparison.

[0095] That is, by comparing the first processed data table and the second processed data table, the execution result of the rule corresponding to the comparison data table queue is obtained.

[0096] In this embodiment, when the comparison data table queue includes at least two first data tables and at least two second data tables, the system obtains the first processed data table corresponding to at least two first data tables and the second processed data table corresponding to at least two second data tables according to the second comparison rule, and then processes the data.

[0097] By comparing the first and second processed data tables, the execution results of the rules corresponding to the comparison data table queue are obtained, which can effectively compare complex business data and improve data migration efficiency.

[0098] In one embodiment, the heterogeneous database comparison result may include multiple data table comparison results. Obtaining the heterogeneous database comparison result corresponding to the current data comparison task based on the first execution result and the second execution result from each of the rule execution results may include the following steps:

[0099] For each rule execution result, the first execution result and the second execution result in the rule execution result are parsed to obtain the first feature value corresponding to the first execution result and the second feature value corresponding to the second execution result; based on the comparison result of the first feature value and the second feature value, the data table comparison result corresponding to the comparison data table queue to which the rule execution result belongs is obtained.

[0100] In specific implementations, such as Figure 3b As shown, SHA-2 encoding calculation can be performed on each comparison execution result file (i.e. rule execution result) to parse the comparison execution result. The SHA-2 encoded value of the comparison execution result file of the data source database (i.e. the first feature value corresponding to the first execution result) is compared with the SHA-2 encoded value of the comparison execution result file of the corresponding imported data in the data import database (i.e. the second feature value corresponding to the second execution result).

[0101] For example, if the SHA-2 encoded values ​​are the same, it can be confirmed that the source data and imported data of the compared data table are completely the same. If the SHA-2 encoded values ​​are different, it can be confirmed that the source data and imported data of the compared data table are not completely the same. That is, based on the comparison results of the first feature value and the second feature value, the data table comparison result corresponding to the comparison data table queue to which the rule execution result belongs can be obtained.

[0102] In this embodiment, by analyzing the first and second execution results of each rule execution result, the first feature value corresponding to the first execution result and the second feature value corresponding to the second execution result are obtained. Then, based on the comparison result of the first feature value and the second feature value, the data table comparison result corresponding to the comparison data table queue to which the rule execution result belongs is obtained, which can improve the accuracy of data verification and comparison.

[0103] In one embodiment, such as Figure 4 The diagram illustrates another method for processing heterogeneous database data. In this embodiment, the method includes the following steps:

[0104] In step 401, migration plan information and actual migration information from the data source database are obtained, and a first set of data tables is determined from the data source database to obtain the comparison range information for the current data comparison task. In step 402, based on the data change information corresponding to each first data table, the type classification result corresponding to each first data table is obtained; different type classification results correspond to different comparison rule information. In step 403, based on the type classification results corresponding to each first data table, multiple comparison rule information is determined. In step 404, combining multiple comparison rule information, the first set of data tables, and the second set of data tables, multiple comparison data table queues are determined. In step 405, multiple comparison data table queues are processed using multi-threading. When a comparison data table queue includes one first data table and one second data table, comparison is performed according to the first comparison rule to obtain the rule execution result corresponding to the comparison data table queue. In step 406, when a comparison data table queue includes at least two first data tables and at least two second data tables, the first processed data table and the second processed data table are compared according to the second comparison rule to obtain the rule execution result corresponding to the comparison data table queue. In step 407, for each rule execution result, the first and second execution results are parsed to obtain the first feature value corresponding to the first execution result and the second feature value corresponding to the second execution result. In step 408, based on the comparison result of the first and second feature values, the data table comparison result corresponding to the comparison data table queue to which the rule execution result belongs is obtained. It should be noted that the specific limitations of the above steps can be found in the specific limitations of a heterogeneous database data processing method described above, and will not be repeated here.

[0105] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0106] Based on the same inventive concept, this application also provides a heterogeneous database data processing apparatus for implementing the heterogeneous database data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the heterogeneous database data processing apparatus provided below can be found in the limitations of the heterogeneous database data processing method described above, and will not be repeated here.

[0107] In one embodiment, such as Figure 5 As shown, a heterogeneous database data processing device is provided, comprising:

[0108] The comparison range acquisition module 501 is used to acquire the comparison range information of the current data comparison task; the comparison range information is used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, wherein the data tables in the first set of data tables correspond one-to-one with the data tables in the second set of data tables, and the first set of data tables is determined according to the data migration situation of the data source database;

[0109] The comparison rule determination module 502 is used to determine the comparison data table queues corresponding to each of the multiple comparison rule information based on the type classification results corresponding to each first data table in the first data table set; each comparison data table queue includes at least one first data table and at least one second data table corresponding to the at least one first data table;

[0110] The rule execution result obtaining module 503 is used to process multiple comparison data table queues through multi-threading to obtain the rule execution result corresponding to each comparison data table queue; the rule execution result includes a first execution result obtained based on at least one first data table and a second execution result obtained based on at least one second data table;

[0111] The comparison result obtaining module 504 is used to obtain the heterogeneous database comparison result corresponding to the current data comparison task based on the first execution result and the second execution result in each of the rule execution results; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database.

[0112] In one embodiment, the comparison range acquisition module 501 includes:

[0113] The migration-related information acquisition submodule is used to acquire migration plan information and actual migration information of the data source database; the migration plan information is used to characterize the data tables to be migrated in the data source database, and the actual migration information is used to characterize the data tables that have been migrated in the data source database at the start of the current data comparison task;

[0114] The comparison range determination submodule is used to determine the first set of data tables from the data source database based on the migration plan information and the actual migration information, and to obtain the comparison range information of the current data comparison task.

[0115] In one embodiment, the comparison rule determination module 502 includes:

[0116] The type classification result acquisition submodule is used to obtain the type classification result corresponding to each of the first data tables based on the data change information corresponding to each of the first data tables; different type classification results correspond to different comparison rule information;

[0117] The comparison rule information determination submodule is used to determine multiple comparison rule information based on the type classification results corresponding to each of the first data tables;

[0118] The comparison data table queue is obtained by a sub-module, which combines the multiple comparison rule information, the first data table set, and the second data table set to determine the multiple comparison data table queues.

[0119] In one embodiment, the comparison data table queue obtaining submodule includes:

[0120] The data table determination unit is configured to determine at least one first data table from the first data table set for each comparison rule information, and to determine at least one second data table corresponding to the at least one first data table from the second data table set.

[0121] The comparison data table queue obtaining unit is used to obtain the comparison data table queue corresponding to the comparison rule information based on the at least one first data table and the at least one second data table.

[0122] In one embodiment, the rule execution result obtaining module 503 includes:

[0123] The first comparison rule execution submodule is used to compare the first data table and the second data table according to the first comparison rule when the comparison data table queue includes a first data table and a second data table, and obtain the rule execution result corresponding to the comparison data table queue.

[0124] In one embodiment, the rule execution result obtaining module 503 includes:

[0125] The second comparison rule execution submodule is used to obtain, according to the second comparison rule, the first processed data table corresponding to the at least two first data tables and the second processed data table corresponding to the at least two second data tables when the comparison data table queue includes at least two first data tables and at least two second data tables.

[0126] The execution result acquisition submodule is used to compare the first processed data table and the second processed data table to obtain the rule execution result corresponding to the comparison data table queue.

[0127] In one embodiment, the heterogeneous database comparison result includes comparison results of multiple data tables, and the comparison result obtaining module 504 includes:

[0128] The rule parsing submodule is used to parse the first execution result and the second execution result in each rule execution result to obtain the first feature value corresponding to the first execution result and the second feature value corresponding to the second execution result.

[0129] The feature value comparison submodule is used to obtain the data table comparison result corresponding to the comparison data table queue to which the rule execution result belongs, based on the comparison result of the first feature value and the second feature value.

[0130] Each module in the aforementioned heterogeneous database data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0131] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a heterogeneous database data processing method.

[0132] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0134] Obtain the comparison range information for the current data comparison task; the comparison range information is used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, wherein the data tables in the first set of data tables correspond one-to-one with the data tables in the second set of data tables, and the first set of data tables is determined based on the data migration situation of the data source database;

[0135] Based on the type classification results corresponding to each first data table in the first data table set, a comparison data table queue corresponding to each of the multiple comparison rule information is determined; each comparison data table queue includes at least one first data table and at least one second data table corresponding to the at least one first data table;

[0136] Multiple comparison data table queues are processed by multiple threads to obtain the rule execution result corresponding to each comparison data table queue; the rule execution result includes a first execution result based on at least one first data table and a second execution result based on at least one second data table;

[0137] Based on the first execution result and the second execution result in the execution results of each rule, the heterogeneous database comparison result corresponding to the current data comparison task is obtained; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database.

[0138] In one embodiment, when the processor executes the computer program, it also implements the steps of the heterogeneous database data processing method in the other embodiments described above.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0140] Obtain the comparison range information for the current data comparison task; the comparison range information is used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, wherein the data tables in the first set of data tables correspond one-to-one with the data tables in the second set of data tables, and the first set of data tables is determined based on the data migration situation of the data source database;

[0141] Based on the type classification results corresponding to each first data table in the first data table set, a comparison data table queue corresponding to each of the multiple comparison rule information is determined; each comparison data table queue includes at least one first data table and at least one second data table corresponding to the at least one first data table;

[0142] Multiple comparison data table queues are processed by multiple threads to obtain the rule execution result corresponding to each comparison data table queue; the rule execution result includes a first execution result based on at least one first data table and a second execution result based on at least one second data table;

[0143] Based on the first execution result and the second execution result in the execution results of each rule, the heterogeneous database comparison result corresponding to the current data comparison task is obtained; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database.

[0144] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the heterogeneous database data processing method in the other embodiments described above.

[0145] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0146] Obtain the comparison range information for the current data comparison task; the comparison range information is used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, wherein the data tables in the first set of data tables correspond one-to-one with the data tables in the second set of data tables, and the first set of data tables is determined based on the data migration situation of the data source database;

[0147] Based on the type classification results corresponding to each first data table in the first data table set, a comparison data table queue corresponding to each of the multiple comparison rule information is determined; each comparison data table queue includes at least one first data table and at least one second data table corresponding to the at least one first data table;

[0148] Multiple comparison data table queues are processed by multiple threads to obtain the rule execution result corresponding to each comparison data table queue; the rule execution result includes a first execution result based on at least one first data table and a second execution result based on at least one second data table;

[0149] Based on the first execution result and the second execution result in the execution results of each rule, the heterogeneous database comparison result corresponding to the current data comparison task is obtained; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database.

[0150] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the heterogeneous database data processing method in the other embodiments described above.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0152] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing heterogeneous database data, characterized in that, The method includes: Obtain the comparison range information for the current data comparison task; the comparison range information is used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, wherein the data tables in the first set of data tables correspond one-to-one with the data tables in the second set of data tables, and the first set of data tables is determined based on the data migration situation of the data source database; Based on the type classification results corresponding to each first data table in the first data table set, a comparison data table queue corresponding to each of the multiple comparison rule information is determined; each comparison data table queue includes at least one first data table and at least one second data table corresponding to the at least one first data table; Multiple comparison data table queues are processed by multiple threads to obtain the rule execution result corresponding to each comparison data table queue; the rule execution result includes a first execution result based on at least one first data table and a second execution result based on at least one second data table; Based on the first execution result and the second execution result in the execution results of each rule, the heterogeneous database comparison result corresponding to the current data comparison task is obtained; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database.

2. The method according to claim 1, characterized in that, The step of obtaining the comparison range information for the current data comparison task includes: Obtain migration plan information and actual migration information of the data source database; the migration plan information is used to characterize the data tables to be migrated in the data source database, and the actual migration information is used to characterize the data tables that have been migrated in the data source database at the start of the current data comparison task; Based on the migration plan information and the actual migration information, a first set of data tables is determined from the data source database to obtain the comparison range information for the current data comparison task.

3. The method according to claim 1, characterized in that, The step of determining the comparison data table queues corresponding to each of the multiple comparison rule information based on the type classification results of each first data table in the first data table set includes: Based on the data change information corresponding to each of the first data tables, the type classification results corresponding to each of the first data tables are obtained; different type classification results correspond to different comparison rule information; Based on the type classification results corresponding to each of the first data tables, multiple comparison rule information is determined; By combining the multiple comparison rule information, the first data table set, and the second data table set, the multiple comparison data table queues are determined.

4. The method according to claim 3, characterized in that, The step of combining the multiple comparison rule information, the first data table set, and the second data table set to determine the multiple comparison data table queues includes: For each comparison rule, at least one first data table is determined from the first data table set, and at least one second data table corresponding to the at least one first data table is determined from the second data table set; Based on the at least one first data table and the at least one second data table, a comparison data table queue corresponding to the comparison rule information is obtained.

5. The method according to claim 1, characterized in that, The process of obtaining the rule execution results corresponding to each of the comparison data table queues includes: When the comparison data table queue includes a first data table and a second data table, the first data table and the second data table are compared according to the first comparison rule to obtain the rule execution result corresponding to the comparison data table queue.

6. The method according to claim 1, characterized in that, The process of obtaining the rule execution results corresponding to each of the comparison data table queues includes: When the comparison data table queue includes at least two first data tables and at least two second data tables, the first processed data table corresponding to the at least two first data tables and the second processed data table corresponding to the at least two second data tables are obtained according to the second comparison rule. By comparing the first processed data table and the second processed data table, the rule execution results corresponding to the comparison data table queue are obtained.

7. The method according to any one of claims 1 to 6, characterized in that, The heterogeneous database comparison result includes multiple data table comparison results. The step of obtaining the heterogeneous database comparison result corresponding to the current data comparison task based on the first execution result and the second execution result among the execution results of each rule includes: For each rule execution result, the first execution result and the second execution result are analyzed to obtain the first feature value corresponding to the first execution result and the second feature value corresponding to the second execution result; Based on the comparison results of the first feature value and the second feature value, the comparison results of the data table corresponding to the comparison data table queue to which the rule execution result belongs are obtained.

8. A heterogeneous database data processing device, characterized in that, The device includes: The comparison range acquisition module is used to acquire the comparison range information of the current data comparison task; the comparison range information is used to indicate the first set of data tables in the data source database to be compared and the second set of data tables in the data import database, wherein the data tables in the first set of data tables correspond one-to-one with the data tables in the second set of data tables, and the first set of data tables is determined according to the data migration situation of the data source database; The comparison rule determination module is used to determine the comparison data table queues corresponding to each of the multiple comparison rule information based on the type classification results corresponding to each first data table in the first data table set; each comparison data table queue includes at least one first data table and at least one second data table corresponding to the at least one first data table; The rule execution result obtaining module is used to process multiple comparison data table queues through multi-threading to obtain the rule execution result corresponding to each comparison data table queue; the rule execution result includes a first execution result obtained based on at least one first data table and a second execution result obtained based on at least one second data table; The comparison result obtaining module is used to obtain the heterogeneous database comparison result corresponding to the current data comparison task based on the first execution result and the second execution result in the execution results of each rule; the heterogeneous database comparison result is used to characterize the data migration differences between the data source database and the data import database.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images