Heterogeneous database migration and data comparison method, device, equipment and storage medium

Abstract

The present invention discloses a heterogeneous database migration and data comparison method, apparatus, device and storage medium. The method comprises: creating a multi-level thread pool, the multi-level thread pool comprising a parent thread pool and a child thread pool; executing a task in the parent thread pool, analyzing the data to be exported in the first database according to the library and table partitioning strategy, obtaining splitting conditions and data features, and generating subtasks; executing the subtasks in the child thread pool, exporting the data to be exported from the first database according to the splitting conditions; transcoding the exported data and importing it into the corresponding second database; and using a cosine similarity algorithm to compare and correct the data exported from the first database and the data imported into the second database. Compared with the prior art, the embodiments of the present invention solve the problem that database migration cannot balance efficiency and cost, and solve the problem that data comparison cannot balance efficiency and data migration quality.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a heterogeneous database migration and data comparison method, device, equipment and storage medium. Background Art

[0002] With the current state of business development and technology stack transformation, commercial banks and other enterprises are faced with the need to migrate data between heterogeneous databases (non-identical databases). Because commercial banks rely heavily on various customer data, data consistency before and after migration is crucial. Therefore, designing heterogeneous database data migration and efficient data consistency comparison mechanisms is crucial to facilitate data migration in banking systems and ensure strong consistency before and after migration.

[0003] In the existing technology, there are two approaches to database migration. One involves deploying the system on a Hadoop cluster, which is large-scale and costly, and wastes resources when migrating smaller amounts of data. The other involves importing and exporting a single table as a single task node. This approach cannot parallelize single-table imports and exports, and its efficiency is limited when the data volume is large. There are two main approaches to comparing data before and after migration: one is to compare rows, which is a rough comparison and can easily overlook anomalies; the other is to compare each row individually, which is inefficient. Summary of the Invention

[0004] The present invention provides a heterogeneous database migration and data comparison method, apparatus, device and storage medium to solve the problem that database migration cannot strike a balance between efficiency and cost, and solves the problem that data comparison cannot strike a balance between efficiency and data migration quality.

[0005] According to one aspect of the present invention, a heterogeneous database migration and data comparison method is provided, comprising:

[0006] Create a multi-level thread pool, the multi-level thread pool including a parent thread pool and a child thread pool;

[0007] Executing tasks in the parent thread pool, analyzing the data to be exported in the first database according to the library and table sharding strategy, obtaining splitting conditions and data features, and generating subtasks;

[0008] Executing the subtask in the child thread pool, and exporting the data to be exported from the first database according to the splitting condition;

[0009] Transcode the exported data and import it into the corresponding second database;

[0010] The cosine similarity algorithm is used to compare and correct the data exported from the first database and the data imported from the second database.

[0011] According to another aspect of the present invention, a heterogeneous database migration and data comparison device is provided, comprising:

[0012] A thread pool creation module is used to create a multi-level thread pool, wherein the multi-level thread pool includes a parent thread pool and a child thread pool;

[0013] A subtask generation module is used to execute tasks in the parent thread pool, analyze the data to be exported in the first data according to the library and table sharding strategy, obtain splitting conditions and data characteristics, and generate subtasks;

[0014] A data splitting and exporting module, configured to execute the subtask in the child thread pool and export the data to be exported from the first database according to the splitting condition;

[0015] A data import module, used to transcode the exported data and import it into the corresponding second database;

[0016] The data comparison and correction module is used to compare and correct the data exported from the first database and the data imported from the second database using a cosine similarity algorithm.

[0017] According to another aspect of the present invention, an electronic device is provided, comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the heterogeneous database migration and data comparison method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the heterogeneous database migration and data comparison method described in any embodiment of the present invention when executed.

[0022] In an embodiment of the present invention, the use of a multi-level thread pool can realize multi-threaded parallel export of data from a first database (for example, an Oracle database), and then multi-threaded concurrent transcoding of the split data and importing it into a second database (for example, a MySQL database). Among them, the multi-level thread pool task scheduling method provides an automated data migration method from the first database to the second database, reducing the risk of misoperation that may exist in manual intervention. In addition, the embodiment of the present invention integrates an automated data comparison and supplementary import method. By calculating the characteristic vectors of the data modules before and after data migration and using the cosine similarity algorithm, the data difference module is quickly located, ensuring the sensitive characteristics of data consistency during the system data migration process, while improving the data comparison efficiency, and finally supplementing the imported data to improve data consistency and ensure the quality of data migration. The embodiment of the present invention rationally plans the migration tasks and uses the multi-level thread pool method to improve the migration efficiency by making full use of the concurrency in each step. Moreover, when performing non-massive data migration, the embodiment of the present invention does not need to deploy a large-scale data migration and comparison system. While saving the cost of migration, it efficiently utilizes resources and speeds up the migration efficiency. In summary, the embodiments of the present invention solve the problem that database migration cannot strike a balance between efficiency and cost, and solve the problem that data comparison cannot strike a balance between efficiency and data migration quality.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a heterogeneous database migration and data comparison method provided by an embodiment of the present invention;

[0026] Figure 2 A flowchart of a system startup provided by an embodiment of the present invention;

[0027] Figure 3 A flowchart of a parent thread pool executing a task provided by an embodiment of the present invention;

[0028] Figure 4 A flowchart of data splitting and exporting in a sub-task executed by a child thread pool according to an embodiment of the present invention;

[0029] Figure 5 A flowchart of data transcoding and importing in a subtask executed by a child thread pool according to an embodiment of the present invention;

[0030] Figure 6 A flowchart of data comparison and correction in a sub-task executed by a sub-thread pool provided in an embodiment of the present invention;

[0031] Figure 7 A flowchart of a thread scanning sequence of a child thread pool provided by an embodiment of the present invention;

[0032] Figure 8 A schematic diagram of the structure of a heterogeneous database migration and data comparison device provided by an embodiment of the present invention;

[0033] Figure 9 A schematic diagram of the relationship between a heterogeneous database migration and data comparison device and a banking system provided by an embodiment of the present invention;

[0034] Figure 10 A schematic structural diagram of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Figure 1This is a flow chart of a heterogeneous database migration and data comparison method provided by an embodiment of the present invention. This embodiment is applicable to data migration between heterogeneous databases of commercial banks. The method can be executed by a heterogeneous database migration and data comparison device. The heterogeneous database migration and data comparison device can be implemented in the form of hardware and / or software. The heterogeneous database migration and data comparison device can be configured in a server or client. Figure 1 As shown, the method includes the following steps:

[0038] S110. Create a multi-level thread pool, where the multi-level thread pool includes a parent thread pool and a child thread pool.

[0039] S120. Execute tasks in the parent thread pool, analyze the data to be exported in the first database according to the library and table sharding strategy, obtain splitting conditions and data features, and generate subtasks.

[0040] The first database may be, for example, an Oracle database.

[0041] S130: Execute subtasks in the child thread pool and export the data to be exported from the first database according to the splitting conditions.

[0042] S140: Transcode the exported data and import it into the corresponding second database.

[0043] The transcoded data format may be, for example, utf8 encoding, and the second database may be, for example, a mysql database.

[0044] S150 , using a cosine similarity algorithm, comparing and correcting the data exported from the first database and the data imported from the second database.

[0045] Among them, the cosine similarity algorithm refers to the use of the cosine value of the angle between two vectors in a vector space as a measure of the size of the difference between two individuals. The closer the cosine value is to 1, the closer the angle is to 0 degrees, indicating that the two vectors are more similar. The closer the cosine value is to 0, the closer the angle is to 90 degrees, indicating that the two vectors are less similar. Specifically, the eigenvector of the data module in the first database is calculated to obtain the first eigenvector; the eigenvector of the data module in the second database is calculated to obtain the second eigenvector; the cosine similarity calculation of the first eigenvector and the second eigenvector can be performed to obtain the similarity of the data modules in the first database and the second database. By comparing the first eigenvector and the second eigenvector, the inconsistent data modules are located, and the inconsistent data modules are compared one by one for data supplementation and import.

[0046] In an embodiment of the present invention, the use of a multi-level thread pool can realize multi-threaded parallel export of data from a first database (for example, an Oracle database), and then multi-threaded concurrent transcoding of the split data and importing it into a second database (for example, a MySQL database). Among them, the multi-level thread pool task scheduling method provides an automated data migration method from the first database to the second database, reducing the risk of misoperation that may exist in manual intervention. In addition, the embodiment of the present invention integrates an automated data comparison and supplementary import method. By calculating the characteristic vectors of the data modules before and after data migration and using the cosine similarity algorithm, the data difference module is quickly located, ensuring the sensitive characteristics of data consistency during the system data migration process, while improving the data comparison efficiency, and finally supplementing the imported data to improve data consistency and ensure the quality of data migration. The embodiment of the present invention rationally plans the migration tasks and uses the multi-level thread pool method to improve the migration efficiency by making full use of the concurrency in each step. Moreover, when performing non-massive data migration, the embodiment of the present invention does not need to deploy a large-scale data migration and comparison system. While saving the cost of migration, it efficiently utilizes resources and speeds up the migration efficiency. In summary, the embodiments of the present invention solve the problem that database migration cannot strike a balance between efficiency and cost, and solve the problem that data comparison cannot strike a balance between efficiency and data migration quality.

[0047] In the above embodiments, optionally, when executing tasks and subtasks of data migration and import, it is also necessary to change the status of tasks and subtasks in real time, so as to facilitate the smooth execution of tasks and subtasks. Specifically, the status of tasks includes: one of the status to be exported, the status in scanning, and the status of completed import; the status of subtasks includes: one of the status to be split, the status in splitting, the status to be imported, the status in importing, the status of completed import, the status in confirming, and the status of completed confirmation.

[0048] Figure 2 This is a flowchart of a system startup provided by an embodiment of the present invention. Figure 2 Based on the above embodiments, optionally, creating a multi-level thread pool is the last step of the system startup process. The system startup process specifically includes:

[0049] S111. Read the configuration file.

[0050] The configuration file content may include, for example, data tables that need to be exported, database and table sharding strategies, and other migration information.

[0051] S112: Generate tasks according to the configuration file.

[0052] For example, the data tables to be exported, the database and table sharding strategies, and other migration information generation tasks are performed as needed.

[0053] S113: Write the task into the export task table and set the task status to pending export.

[0054] S114. Create a multi-level thread pool.

[0055] The embodiment of the present invention implements the system startup steps through S111-S114. Before data splitting and exporting, the data tables, database and table sharding strategies and other migration information to be exported are read into the export task table by reading the configuration file content, setting the task to the pending export state, and creating a multi-level thread pool.

[0056] Figure 3 This is a flowchart of a parent thread pool executing a task provided by an embodiment of the present invention. Figure 3 Based on the above embodiments, optionally, executing a task in the parent thread pool includes the following steps:

[0057] S121. Scan and export the task table regularly.

[0058] Among them, multiple threads in the parent thread pool can run in parallel.

[0059] S122: Find the tasks in the export task table that are in the to-be-exported state.

[0060] S123: Change the status of the found task to scanning.

[0061] S124. According to the database and table sharding strategy, the data to be exported in the first data is analyzed to obtain splitting conditions and data features.

[0062] S125. Generate subtasks based on the splitting conditions and data features.

[0063] S126: Insert the subtask into the split task table, and set the status of the subtask to the pending split status.

[0064] S127. After the comparison and correction are completed, the task status is changed to Scan Completed.

[0065] In the embodiment of the present invention, through S121-S127, multiple threads in the parent thread pool periodically scan the export task table. The execution process of one of the threads is to find the task in the to-be-exported state, modify the task status to the scanning state and then preempt the task, perform data scanning and analysis, divide the task into multiple subtasks according to the splitting conditions, fill in the split task table, set the subtask status to the to-be-split state, and after the scan is completed, modify the task status in the export task table to the scan completed state.

[0066] Continue to see Figure 3 Optionally, after S127, it also includes S128, hibernation.

[0067] Figure 4 This is a flowchart of a sub-thread pool executing data splitting and exporting in a subtask according to an embodiment of the present invention. Figure 4 Based on the above embodiments, optionally, exporting the data to be exported from the first database includes the following steps:

[0068] S131. The child thread pool scans the split task table.

[0069] S132: Find the subtasks in the split task table that are in a state of being split.

[0070] S133: Change the status of the found subtask to "being split".

[0071] S134. Export the data to be exported from the first database according to the splitting condition.

[0072] S135: Change the status of the subtask of exporting data to pending import.

[0073] It can be seen that through S131-S135, the child thread pool scans the split task table, changes the subtask status to the splitting state, and then splits and exports the data according to the database and table sharding strategy. After completing the task, the subtask status is changed to the pending import state.

[0074] Continue to see Figure 4 Optionally, after S135, the process further includes S136: hibernation.

[0075] Figure 5 This is a flowchart of a sub-task execution data transcoding import in a child thread pool according to an embodiment of the present invention. Figure 5 Based on the above embodiments, optionally, the exported data is transcoded and imported into the corresponding second database, including the following steps:

[0076] S141. Scan and split the task table.

[0077] S142: If there is no subtask in the split task table that is in the to-be-split state, search for a subtask in the to-be-imported state in the split task table.

[0078] S143. Change the status of the found subtask to "importing".

[0079] S144: Transcode the exported data and import it into the corresponding second database.

[0080] S145. Change the status of the subtask that has been imported to completed import.

[0081] It can be seen that through S141-S145, after the child thread pool fails to find the subtask in the state to be split, it looks for the subtask in the state to be imported, changes the subtask state to the preempted task in execution, finds the corresponding data, transcodes it and imports it into the second database (for example, MySQL database). After the import is completed, the subtask state is changed to completed import.

[0082] Continue to see Figure 5 Optionally, after S145, it also includes S146, hibernation.

[0083] Figure 6 This is a flowchart of data comparison and correction in a sub-task executed by a child thread pool according to an embodiment of the present invention. Figure 6 Based on the above embodiments, optionally, performing comparison and correction on the data exported from the first database and the data imported from the second database includes the following steps:

[0084] S151, scan and split the task table.

[0085] S152: If there is no subtask in the pending import state in the split task table, search for a subtask in the completed import state in the split task table.

[0086] S153: Change the status of the found subtask to "Confirming".

[0087] S154: Find the corresponding data module in the first database and the second database.

[0088] S155 : Calculate the eigenvector of the data module in the first database to obtain a first eigenvector.

[0089] S156. Calculate the eigenvector of the data module in the second database to obtain a second eigenvector.

[0090] Exemplarily, 36-dimensional feature vectors (26 letters and 10 numbers, a total of 36 features) are calculated based on the primary keys of the two data modules, and the feature vector value is the number of occurrences of each feature.

[0091] S157 , performing cosine similarity calculation on the first eigenvector and the second eigenvector to obtain similarity between the data modules in the first database and the second database.

[0092] S158 , determining whether the similarity meets the similarity threshold requirement, if so, executing S15B; otherwise, executing S159 .

[0093] The similarity threshold may be set to 100% or other values, for example.

[0094] S159: Compare the data modules in the first database and the second database one by one.

[0095] S15A. Find the data that needs to be imported and enter the additional data.

[0096] S15B. Change the status of the subtask to completion confirmation.

[0097] It can be seen that through S151-S15B, after the sub-level thread pool cannot find the data in the waiting-to-import state, it looks for the subtask in the completed import state, modifies the state to confirming and then preempts the task, finds the corresponding data module in the first database (for example, Oracle database) and the second database (for example, MySQL database), calculates the feature vector, calculates the similarity according to the cosine algorithm, and determines whether the subtask is completed or needs to be supplemented and imported according to the set similarity threshold. After confirming that no supplementary import is required, the subtask status is set to completion confirmation. If the subtask similarity threshold does not meet the requirements, the thread compares the data modules one by one, finds the data that needs to be supplemented and imported, and corrects it.

[0098] Continue to see Figure 6 Optionally, after S15B, it also includes S15C, hibernation.

[0099] Figure 7 This is a flowchart of a thread scanning sequence of a child thread pool provided by an embodiment of the present invention. Figure 7 Based on the above embodiments, optionally, the child thread pool thread scanning step includes:

[0100] S210: Scan and split the task table.

[0101] S220 , searching for subtasks in the split task table whose status is to be split; if a subtask to be split is found, executing S230 ; otherwise executing S240 .

[0102] S230: Execute the step of splitting and exporting data.

[0103] S240 , searching for subtasks in the split task table whose status is to be imported; if a subtask to be imported is found, executing S250 ; otherwise executing S260 .

[0104] S250: Execute the step of data transcoding and importing.

[0105] S260 , searching for a subtask in the split task table whose status is "Import Completed"; if a subtask is found, executing S270 ; otherwise, the thread ends.

[0106] S270: Execute data comparison and correction steps.

[0107] It can be seen that through S210-S270, the scanning of the child thread pool is completed.

[0108] An embodiment of the present invention further provides a heterogeneous database migration and data comparison device, which can be executed by software and / or hardware. Figure 8 This is a schematic diagram of the structure of a heterogeneous database migration and data comparison device provided by an embodiment of the present invention. Figure 8 , heterogeneous database migration and data comparison device includes:

[0109] The thread pool creation module 310 is used to create a multi-level thread pool, which includes a parent thread pool and a child thread pool;

[0110] The subtask generation module 320 is used to execute tasks in the parent thread pool, analyze the data to be exported in the first data according to the library and table sharding strategy, obtain splitting conditions and data characteristics, and generate subtasks;

[0111] The data splitting and exporting module 330 is used to execute subtasks in the child thread pool and export the data to be exported from the first database according to the splitting conditions;

[0112] The data import module 340 is used to transcode the exported data and import it into the corresponding second database;

[0113] The data comparison and correction module 350 is used to compare and correct the data exported from the first database and the data imported from the second database using a cosine similarity algorithm.

[0114] In an embodiment of the present invention, a multi-level thread pool task scheduling method is used through a thread pool creation module 310, a subtask generation module 320, a data splitting and exporting module 330, a data import module 340, and a data comparison and correction module 350 to provide an automated data migration method from a first database to a second database, thereby reducing the risk of misoperation that may exist due to manual intervention. In addition, an embodiment of the present invention integrates an automated data comparison and supplementary import method. By calculating the characteristic vectors of the data modules before and after data migration and using the cosine similarity algorithm, the data difference module is quickly located, ensuring the relatively sensitive characteristics of data consistency during the system data migration process, while improving the data comparison efficiency, and ultimately supplementing the imported data to improve data consistency and ensure the quality of data migration. The embodiment of the present invention rationally plans the migration tasks and uses a multi-level thread pool method to improve the migration efficiency by making full use of the concurrency in each step. Moreover, when performing non-massive data migration, the embodiment of the present invention does not need to deploy a large-scale data migration and comparison system. While saving the cost of migration, it efficiently utilizes resources and speeds up the migration efficiency. In summary, the embodiments of the present invention solve the problem that database migration cannot strike a balance between efficiency and cost, and solve the problem that data comparison cannot strike a balance between efficiency and data migration quality.

[0115] Optionally, the task status includes one of: a pending export status, a scanning status, and an import completed status;

[0116] The status of a subtask includes one of: waiting to be split, splitting, waiting to be imported, importing, import completed, confirming, and completed confirmed.

[0117] Optionally, the subtask generation module 320 is specifically configured to execute tasks in the parent thread pool, including:

[0118] Scan the export task table regularly to find tasks in the export task table that are in the pending export state;

[0119] Change the status of the found task to scanning;

[0120] According to the database and table sharding strategy, the data to be exported in the first data is analyzed to obtain splitting conditions and data features;

[0121] Generate subtasks based on splitting conditions and data features;

[0122] Insert the subtask into the split task table and set the status of the subtask to be split;

[0123] After the comparison and correction are completed, the task status is changed to Scan Completed.

[0124] Optionally, the data splitting and exporting module 330 is specifically configured to:

[0125] The child thread pool scans the split task table and finds the subtasks in the split task table that are in the state of being split;

[0126] Change the status of the found subtask to being split;

[0127] Exporting the data to be exported from the first database according to the splitting conditions;

[0128] Change the status of the data export subtask to pending import.

[0129] Optionally, the data import module 340 is specifically configured to:

[0130] If there is no subtask in the split task table that is in the state of being split, the child thread pool scans the split task table and finds the subtask in the state of being imported in the split task table;

[0131] Change the status of the found subtask to "Importing";

[0132] Transcode the exported data and import it into the corresponding second database;

[0133] Change the status of the subtask that has completed import to completed import.

[0134] Optionally, the data comparison and correction module 350 is specifically configured to:

[0135] If there is no subtask in the pending import state in the split task table, the child thread pool scans the split task table and finds the subtask in the completed import state in the split task table;

[0136] Change the status of the found subtask to confirming;

[0137] Finding corresponding data modules in the first database and the second database;

[0138] Calculating the characteristic vector of the data module in the first database to obtain a first characteristic vector; calculating the characteristic vector of the data module in the second database to obtain a second characteristic vector;

[0139] Performing cosine similarity calculation on the first eigenvector and the second eigenvector to obtain similarity between the data modules in the first database and the second database;

[0140] If the similarity meets the similarity threshold requirement, the status of the subtask is changed to completion confirmation;

[0141] Otherwise, the data modules in the first database and the second database are compared one by one, and the data that needs to be supplemented and imported is found to be supplemented and entered, and the status of the subtask is changed to completion confirmation.

[0142] Optionally, the thread pool creation module 310 is further specifically configured to:

[0143] Read the configuration file;

[0144] Generate tasks based on configuration files;

[0145] Write the task to the export task table and set the task status to pending export.

[0146] Figure 9 A schematic diagram of the relationship between a heterogeneous database migration and data comparison device and a banking system provided by an embodiment of the present invention. Figure 9 Based on the above embodiments, heterogeneous database migration and data comparison device 440 is optionally applied to a banking system, where the first database is an Oracle database 420 and the second database is a MySQL database 430. Banking system 410 performs data transfer with Oracle database 420 and with MySQL database 430. Oracle database 420 performs data migration and comparison via heterogeneous database migration and data comparison device 440.

[0147] Figure 10 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0148] like Figure 10 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0149] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0150] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the heterogeneous database migration and data comparison methods.

[0151] In some embodiments, the heterogeneous database migration and data comparison method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the heterogeneous database migration and data comparison method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the heterogeneous database migration and data comparison method in any other appropriate manner (for example, by means of firmware).

[0152] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0157] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0158] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0159] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A heterogeneous database migration and data comparison method, characterized in that: include: Create a multi-level thread pool, the multi-level thread pool including a parent thread pool and a child thread pool; Executing tasks in the parent thread pool, analyzing the data to be exported in the first database according to the library and table sharding strategy, obtaining splitting conditions and data features, and generating subtasks; Executing the subtask in the child thread pool, and exporting the data to be exported from the first database according to the splitting condition; Transcode the exported data and import it into the corresponding second database; Using a cosine similarity algorithm, comparing and correcting the data exported from the first database and the data imported from the second database; The status of the subtask includes one of: waiting to be split, splitting, waiting to be imported, importing, completed import, confirming, and completed confirmed; The comparing and correcting the data exported from the first database and the data imported from the second database includes: If there is no subtask in the pending import state in the split task table, the child thread pool scans the split task table and finds the subtask in the completed import state in the split task table; Modify the status of the found subtask to being confirmed; Finding corresponding data modules in the first database and the second database; Calculating the characteristic vector of the data module in the first database to obtain a first characteristic vector; calculating the characteristic vector of the data module in the second database to obtain a second characteristic vector; Performing cosine similarity calculation on the first feature vector and the second feature vector to obtain similarity between data modules in the first database and the second database; If the similarity meets the similarity threshold requirement, the status of the subtask is changed to completion confirmation; Otherwise, the data modules in the first database and the second database are compared one by one, and the data that needs to be supplemented and imported is found to be supplemented and entered, and the status of the subtask is modified to completion confirmation.

2. The heterogeneous database migration and data comparison method according to claim 1, characterized in that: The status of the task includes one of: a pending export status, a scanning status, and an import completed status.

3. The heterogeneous database migration and data comparison method according to claim 2, characterized in that: Executing tasks in the parent thread pool includes: Scan the export task table regularly to find the tasks in the export task table that are in the pending export state; Modify the status of the found task to scanning; According to the database and table sharding strategy, the data to be exported in the first data is analyzed to obtain splitting conditions and data features; generating subtasks according to the splitting conditions and the data features; Insert the subtask into the split task table and set the status of the subtask to be split; After the comparison and correction are completed, the status of the task is changed to scan completed.

4. The heterogeneous database migration and data comparison method according to claim 2, characterized in that: Exporting the data to be exported from the first database includes: The child thread pool scans the split task table to find the subtasks in the split task table that are in the to-be-split state; Modify the status of the found subtask to being split; exporting the data to be exported from the first database according to the splitting condition; The status of the subtask of exporting data is changed to pending import.

5. The heterogeneous database migration and data comparison method according to claim 2, characterized in that: Execute the transcoding of the exported data and import it into the corresponding second database, including: If there is no subtask in the split task table that is in the to-be-split state, the child thread pool scans the split task table to find the subtask in the to-be-imported state in the split task table; Modify the status of the found subtask to importing; Transcoding the exported data and importing it into the corresponding second database; The status of the subtask whose import is completed is changed to import completed.

6. The heterogeneous database migration and data comparison method according to claim 1, characterized in that: Before creating a multi-level thread pool, also include: Read the configuration file; Generate tasks according to the configuration file; The task is written into the export task table, and the status of the task is set to be exported.

7. A heterogeneous database migration and data comparison device, characterized in that: include: A thread pool creation module is used to create a multi-level thread pool, wherein the multi-level thread pool includes a parent thread pool and a child thread pool; A subtask generation module is used to execute tasks in the parent thread pool, analyze the data to be exported in the first database according to the library and table sharding strategy, obtain splitting conditions and data characteristics, and generate subtasks; A data splitting and exporting module, configured to execute the subtask in the child thread pool and export the data to be exported from the first database according to the splitting condition; A data import module, used to transcode the exported data and import it into the corresponding second database; a data comparison and correction module, configured to compare and correct the data exported from the first database and the data imported from the second database using a cosine similarity algorithm; The status of the subtask includes one of: waiting to be split, splitting, waiting to be imported, importing, completed import, confirming, and completed confirmed; The data comparison and correction module is specifically used to: If there is no subtask in the pending import state in the split task table, the child thread pool scans the split task table and finds the subtask in the completed import state in the split task table; Modify the status of the found subtask to being confirmed; Finding corresponding data modules in the first database and the second database; Calculating the characteristic vector of the data module in the first database to obtain a first characteristic vector; calculating the characteristic vector of the data module in the second database to obtain a second characteristic vector; Performing cosine similarity calculation on the first feature vector and the second feature vector to obtain similarity between data modules in the first database and the second database; If the similarity meets the similarity threshold requirement, the status of the subtask is changed to completion confirmation; Otherwise, the data modules in the first database and the second database are compared one by one, and the data that needs to be supplemented and imported is found to be supplemented and entered, and the status of the subtask is modified to completion confirmation.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the heterogeneous database migration and data comparison method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the heterogeneous database migration and data comparison method according to any one of claims 1 to 6 when executed.

Images