Data warehouse relocation method and device, equipment and medium

Abstract

The invention provides a data warehouse relocation method and device, equipment and a medium. The method comprises the steps of constructing a target database based on metadata of a source database; migrating data to be migrated in the source database to the target database; performing data verification on the to-be-migrated data by adopting a preset verification algorithm; wherein the preset verification algorithm comprises a first verification algorithm and a second verification algorithm, the first verification algorithm comprises an LCS algorithm and a DIFF tool, and the second verification algorithm comprises a Rabin-Karp algorithm and the LCS algorithm; and completing the migration of the to-be-migrated data under the condition that the data verification is passed. By adopting the method provided by the embodiment of the invention, the efficiency and accuracy of the data migration method can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a data warehouse migration method, device, equipment and medium. Background Art

[0002] In the related art, when processing the migration of TB (Terabyte)-level data warehouses, since the service performance is often unable to meet the data verification and migration of TB-level data warehouses, it usually takes several days of business interruption to complete the data warehouse migration work. This method will result in a long data migration time and a long business interruption time, which will lead to low migration efficiency. Summary of the invention

[0003] The present disclosure provides a data warehouse migration method, device, equipment and medium. The technical solution of the present disclosure is as follows:

[0004] In a first aspect, the present disclosure provides a data warehouse migration method, comprising:

[0005] Build the target database based on the metadata of the source database;

[0006] Migrating the data to be migrated in the source database to the target database;

[0007] Using a preset verification algorithm to perform data verification on the data to be migrated; wherein the preset verification algorithm includes a first verification algorithm and a second verification algorithm, the first verification algorithm includes an LCS algorithm and a DIFF tool, and the second verification algorithm includes a Rabin-Karp algorithm and the LCS algorithm;

[0008] When the data verification passes, the migration of the data to be migrated is completed.

[0009] In a possible implementation, the step of constructing a target database based on metadata of a source database includes:

[0010] Obtain metadata of the source database; wherein the metadata at least includes table structure, column definition, index, stored procedure, and trigger;

[0011] Parsing the metadata by using preset parsing rules to obtain code generation information; wherein the code generation information is used to generate code for a target database, including at least a table name, a column name, a data type, and a constraint condition;

[0012] Generate the code of the target database based on the code generation information and the code template of the preset data warehouse;

[0013] The target database is constructed based on the code of the target database.

[0014] In a possible implementation manner, migrating the data to be migrated in the source database to the target database includes:

[0015] Capturing the data changes in the source database in real time through the trigger to obtain incremental data;

[0016] The incremental data is determined as the data to be migrated, and is migrated to the target database.

[0017] In a possible implementation, determining the incremental data as the data to be migrated and migrating it to the target database includes:

[0018] Saving changes to data acquired through the trigger to a log file;

[0019] Converting the log file into an XML document;

[0020] Migrate the XML document to the target database.

[0021] In a possible implementation, the using a preset verification algorithm to perform data verification on the to-be-migrated data includes:

[0022] In the case where the data to be migrated meets a preset condition, the LCS algorithm and the DIFF tool are used to perform consistency check on the data to be migrated; wherein the preset condition includes at least one of the following: the amount of the data to be migrated is less than a preset amount of data, and the data to be migrated includes a character string;

[0023] When the data to be migrated does not meet a preset condition, the Rabin-Karp algorithm and the LCS algorithm are used to perform consistency check on the data to be migrated.

[0024] In a possible implementation, the using the LCS algorithm and the DIFF tool to perform consistency check on the data to be migrated includes:

[0025] Identifying the longest common subsequence between the data to be migrated and the migrated data by using the LCS algorithm; wherein the migrated data is data obtained after migrating the data to be migrated in the source database to the target database;

[0026] The DIFF tool is used to perform a consistency check on the longest common subsequence to obtain a check result.

[0027] In a possible implementation, the using the Rabin-Karp algorithm and the LCS algorithm to perform consistency check on the data to be migrated includes:

[0028] Using the Rabin-Karp algorithm to screen the data to be migrated, and locate a character string containing a specific pattern;

[0029] The LCS algorithm is used to perform consistency check on the character string of the specific pattern to obtain a check result.

[0030] In a second aspect, the present disclosure provides a data warehouse migration device, comprising:

[0031] A construction module, used for constructing a target database based on metadata of a source database;

[0032] A migration module, used to migrate the data to be migrated in the source database to the target database;

[0033] A verification module, used to perform data verification on the data to be migrated by using a preset verification algorithm; wherein the preset verification algorithm includes a first verification algorithm and a second verification algorithm, the first verification algorithm includes an LCS algorithm and a DIFF tool, and the second verification algorithm includes a Rabin-Karp algorithm and the LCS algorithm;

[0034] The migration module is used to complete the migration of the data to be migrated if the data verification passes.

[0035] In a third aspect, the present disclosure provides an electronic device, including:

[0036] processor;

[0037] a memory for storing instructions executable by the processor;

[0038] The processor is configured to execute the instructions to implement the method described in the first aspect.

[0039] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0040] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the method described in the first aspect is implemented.

[0041] The technical solution of the embodiment of the present disclosure brings at least the following beneficial effects:

[0042] In the disclosed embodiment, a target database is constructed based on the metadata of a source database; the data to be migrated in the source database is migrated to the target database; a preset verification algorithm is used to perform data verification on the data to be migrated; wherein the preset verification algorithm includes a first verification algorithm and a second verification algorithm, the first verification algorithm includes an LCS algorithm and a DIFF tool, and the second verification algorithm includes a Rabin-Karp algorithm and the LCS algorithm; if the data verification passes, the migration of the data to be migrated is completed. In this way, on the one hand, database construction through metadata can simplify the complex process of cross-database migration and achieve efficient migration of database structure, thereby reducing the workload and error rate of manual programming, improving data migration efficiency, and reducing maintenance costs; on the other hand, data consistency verification using the LCS algorithm and the DIFF tool, or the Rabin-Karp algorithm and the LCS algorithm can ensure data consistency and accuracy, thereby improving the efficiency and accuracy of the data migration method.

[0043] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0045] Figure 1 A flowchart of a data warehouse migration method provided by an embodiment of the present disclosure;

[0046] Figure 2 It is a flowchart of a data warehouse migration method provided by an embodiment of the present disclosure;

[0047] Figure 3 is a flow chart of a data migration method provided by an embodiment of the present disclosure;

[0048] Figure 4 is a flowchart of a data transmission process provided by an embodiment of the present disclosure;

[0049] Figure 5 It is a flowchart of a data verification method provided by an embodiment of the present disclosure;

[0050] Figure 6 It is a flowchart of a data verification method provided by an embodiment of the present disclosure;

[0051] Figure 7 It is a structural schematic diagram of a data warehouse migration device provided by an embodiment of the present disclosure;

[0052] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0055] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0056] The acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations.

[0057] It should be noted that in the embodiments of the present disclosure, there may be existing solutions in the industry such as certain software, components, models, etc., which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0058] In related technologies, with the continuous improvement of domestic scientific and technological strength, domestic software has been widely promoted and applied in various industries. Especially in the telecommunications industry, in order to ensure data security, improve business efficiency and respond to the call for national independent innovation, operators have begun to adopt domestic software and carry out heterogeneous data warehouse cutover and data migration.

[0059] Heterogeneous data warehouse cutover and data migration is a complex and arduous task, which involves multiple aspects such as data integrity, accuracy and system stability. Since different data warehouses may have differences in architecture, data format, storage method, etc., a series of effective methods and measures need to be taken during cutover and migration to ensure the safe migration of data and the smooth transition of the system.

[0060] At present, when dealing with the relocation of TB-level data warehouses, the performance cannot meet the data verification and migration of TB-level data warehouses, and it often takes several days of business interruption to complete the relocation work. This method of data processing is single and cannot cope with structured, semi-structured and unstructured data in heterogeneous data warehouses. When processing complex data types such as JSON (JavaScript Object Notation, JS key-value pair data) documents, log files, pictures, videos, etc., there is a lack of diverse migration strategies and migration technologies. When facing various data quality problems, such as inconsistent data and duplicate data, the lack of effective data problem discovery and problem solving mechanisms not only affects the accuracy of the data warehouse and the availability of data, but may also lead to business interruptions and reduced trust. Traditional data consistency comparison methods often require data comparison one by one, which is impractical for large-scale data sets. In addition, multiple data sources may need to be accessed during the comparison process. The existing data consistency comparison technology often takes several days to complete a data comparison, resulting in slow overall progress of data migration and increased cost of relocation. The current relocation method has at least the following technical problems:

[0061] 1. The relocation preparation time is too long: Due to the huge amount of data, TB-level data warehouses often encounter integrity problems such as primary key conflicts, unique key conflicts, foreign key constraint errors, and consistency problems such as data conversion errors, data loss, and data format mismatches. These problems require us to invest a lot of time and resources to solve them, and we cannot meet the relocation conditions within the expected time. 2. Data consistency is seriously damaged: In the process of data migration, it is very important to maintain data consistency. When dealing with TB-level data warehouses, existing technical solutions often find it difficult to ensure data consistency and accuracy due to the large and complex data volume. This may cause data deviations after migration and affect the accuracy of business decisions. 3. Business interruption time is too long: The complexity and difficulty of relocating TB-level data warehouses, and problems with data integrity and consistency make it difficult to seamlessly switch and migrate TB-level heterogeneous data warehouse data. The business interruption time is too long, which has a serious impact on the normal operation of the enterprise.

[0062] Based on this, the embodiments of the present disclosure provide a data warehouse migration method, device, equipment and medium. By utilizing metadata-driven structural migration technology, cross-database migration can be simplified and programming workload and error rate can be reduced. Trigger technology can ensure real-time lossless migration of data and maintain data consistency and integrity. DIFF (difference detection tool) and LCS (Longest Common Subsequence algorithm) algorithms are used to effectively check data consistency and ensure the reliability of data warehouse migration. Rabin-Karp (Rabin-Karp algorithm) algorithm combined with LCS algorithm is used to improve the migration efficiency and accuracy of TB-level data warehouses, and realize non-perceptible migration without interrupting business. It can effectively improve migration efficiency, reduce costs, shorten data migration preparation time, ensure consistency and accuracy in the data migration process, ensure seamless switching and migration of data between heterogeneous systems, and thus improve the overall efficiency of TB-level data warehouse migration, achieve cost reduction and efficiency improvement, and improve the available value of data. After testing of specific examples, the data warehouse relocation method provided in the embodiment of the present disclosure is used to carry out a TB-level heterogeneous data warehouse relocation project on a specific big data platform, which can save more than 3 million yuan in costs throughout the year, improve relocation efficiency by 33%, and reduce costs by 26%.

[0063] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0064] Figure 1 The flowchart of a data warehouse migration method provided by the embodiment of the present disclosure is as follows. The method can be applied to a server, for example, a single server or a server cluster. Figure 1 As shown, the data warehouse migration method may include the following steps:

[0065] S101, constructing a target database based on metadata of a source database.

[0066] In the embodiments of the present disclosure, it is considered that when performing data migration across data warehouses, due to the differences in database structure, syntax and functions, the code generation and migration process is complicated and time-consuming. Therefore, a metadata-based heterogeneous data warehouse model and code rapid generation technology can be used to complete structural migration and improve data migration efficiency. Exemplarily, when performing data warehouse migration, before performing heterogeneous data warehouse migration, the metadata of the source database can be obtained first, and the metadata of the source database can be used to parse the DDL (Data Definition Language) code of the source database and convert the code into the syntax of the target database. Then, the structural objects are recreated in the target database. In this way, the automatic generation of the target database model and code based on metadata can be achieved.

[0067] In a possible implementation, building a target database based on metadata of a source database includes:

[0068] Obtain metadata of the source database; wherein the metadata at least includes table structure, column definition, index, stored procedure, and trigger;

[0069] Parsing metadata through preset parsing rules to obtain code generation information; wherein the code generation information is used to generate code for a target database, including at least table names, column names, data types, and constraints;

[0070] Generate the code of the target database based on the code generation information and the code template of the preset data warehouse;

[0071] Build the target database based on the target database's code.

[0072] In an embodiment of the present disclosure, when building a target database based on the metadata of a source database, the metadata of the source database can be first obtained. Exemplarily, metadata is first collected and standardized. For example, the metadata required for the target database, including table structure, column definition, index, stored procedure, trigger, etc., can be collected through an API (Application Programming Interface) or tool provided by a DBMS (Database Management System), and the collected metadata is converted into a unified target database model format. Then, metadata parsing can be performed. Exemplarily, metadata can be parsed by preset parsing rules to obtain code generation information. For example, standardized metadata can be parsed using predefined parsing rules to extract key information (code generation information) required for code generation, such as table name, column name, data type, constraint condition, etc. As a specific example, the preset parsing rules involved are mainly used to guide how to extract information from the metadata of an Oracle database and convert it into metadata of a GBase database. These rules include: data type mapping rules, which are used to define a data type mapping table to list Oracle data types and their corresponding GBase data types. During the conversion process, data types are converted according to this mapping table; constraint conversion rules are used to determine how to convert constraints in Oracle (such as primary keys and foreign keys) to corresponding constraints in GBase; syntax adjustment rules are used to write rules for stored procedures and triggers to convert SQL (Structured Query Language) syntax and function calls to meet the requirements of GBase. By setting the above parsing rules, Oracle metadata can be converted to GBase metadata, so as to perform subsequent data conversion, data verification and data cutover.

[0073] After that, code template design and code generation can be performed. Exemplarily, the code of the target database can be generated based on the code generation information and the code template of the preset data warehouse. As an example, code templates for different data warehouses can be designed according to business needs and technical requirements. The parsed metadata is combined with the corresponding code template (the code template of the preset data warehouse) to generate code compatible with the target database system. The generated code includes DAL (Data Access Layer) code, stored procedures, triggers, etc. As a specific example, the process includes: 1) Obtaining table metadata: First, define the get_table_metadata function, input the database type and connection information, and return a list containing table metadata. The information of each table is represented in the form of metadata, including table name, column name, data type, length, and primary key information. 2) Define SQL code templates: For different data warehouse systems (this embodiment mainly involves Oracle and Gbase data warehouses), define corresponding SQL code templates. These templates define how to generate SQL statements for database objects based on metadata, such as SQL statements for creating tables, data access layer (DAL) code, etc. 3) Generate SQL code: define the generate_sql_from_metadata function, input the metadata list and the target database type, select the corresponding SQL code template according to the target database type, traverse each table in the metadata list, fill the table metadata into the template, and generate specific SQL statements. 4) Expand to other database objects: Generate code statements related to other database objects, mainly involving code statements such as stored procedures and triggers. By defining more code templates, specific code statements are generated by referring to different aspects of metadata (such as the logic of stored procedures, the triggering conditions of triggers, etc.). In this way, the efficiency of database design and deployment can be improved, human errors can be reduced, and the accuracy and consistency of database objects can be ensured through automated processes.

[0074] It is understandable that the following processes may also be included:

[0075] Code verification and testing: Verify and test the generated code to ensure its compatibility and correctness with the target database system. Code deployment and maintenance: Deploy the verified code to the target environment and establish a corresponding maintenance mechanism so that the code can be regenerated when the database structure changes.

[0076] In this way, the target database model and code automatic generation technology based on metadata can greatly reduce the workload of manually writing and modifying database code, reduce the human error rate, support multiple heterogeneous database systems, and achieve cross-platform compatibility. When the database structure changes, the code can be quickly regenerated, reducing maintenance costs and greatly reducing the time for heterogeneous data warehouse migration code development.

[0077] S102, migrating the data to be migrated in the source database to the target database.

[0078] In an embodiment of the present disclosure, after building a target database based on the metadata of the source database, data migration can be performed to migrate the data to be migrated in the source database to the target database. Exemplarily, data migration can be performed by migrating multiple batches of real-time incremental data (data to be migrated) based on the combined full and incremental data migration. As an example, real-time data changes can be obtained by using log records and triggers to perform real-time incremental data migration.

[0079] In a possible implementation, migrating the data to be migrated in the source database to the target database includes:

[0080] Capture data changes in the source database in real time through triggers to obtain incremental data;

[0081] The incremental data is identified as the data to be migrated and migrated to the target database.

[0082] In the embodiments of the present disclosure, when migrating the data to be migrated in the source database to the target database, the change of the data in the source database, that is, the change data between the old data and the new data, can be captured in real time through triggers to obtain incremental data, which can be considered as the data to be migrated. Then, the data to be migrated can be migrated to the target database to achieve real-time synchronization of the changed data to the target database. In this way, real-time automatic lossless migration of data can be achieved.

[0083] In a further possible implementation, determining the incremental data as data to be migrated and migrating it to the target database includes:

[0084] Save changes to data obtained through triggers to log files;

[0085] Convert log files to XML documents;

[0086] Migrate the XML document to the target database.

[0087] In an embodiment of the present disclosure, data changes can be saved in real time to a log table, converted into an XML (Extensible Markup Language) document, and relocated to a target data warehouse. Exemplarily, data changes obtained through a trigger can be saved to a log file. Exemplarily, in order to obtain data change information in real time and realize real-time incremental data relocation, a trigger can be created on a synchronization table. The table to be relocated automatically executes a trigger when operations such as INSERT, UPDATE, or DELETE are implemented. The trigger captures the changes in data in the source database in real time, and both old and new data are correctly recorded, such as saved to a log file. Then, the log file can be converted into an XML document, backed up, and the XML document can be migrated to the target database. Exemplarily, a stored procedure or PL / SQL (Procedural Language / SQL) block can be written to query the log table, and the query results can be converted into XML format; then a target database or target table structure is created, and a backup tool such as a data pump (Data Pump) or RMAN (Recovery Manager) is used to back up the XML document (or table containing XML) in the source database to the target database. You can also write PL / SQL code to copy the XML document to the target database through a database link (DBLINK). In this way, by effectively saving log information, when the data in the source data warehouse and the target data warehouse are inconsistent, the data problem can be automatically located based on the log information, reducing manual intervention and achieving automatic lossless data migration.

[0088] As an example, when migrating XML documents to the target database, you can use Java stored procedures (through JDBC, Java Database Connectivity) or external Java applications to access and read XML documents, and send the read XML documents as messages to the OpenJMS message queue. At the other end of the message queue, there are one or more consumer applications that listen to the queue in real time and process the received XML messages. These consumer applications parse the XML messages back into the database table format and load them into the target database or perform other processing. This achieves real-time lossless migration of incremental data.

[0089] S103: Using a preset verification algorithm to perform data verification on the data to be migrated.

[0090] Among them, the preset verification algorithm includes a first verification algorithm and a second verification algorithm. The first verification algorithm includes an LCS algorithm and a DIFF tool, and the second verification algorithm includes a Rabin-Karp algorithm and an LCS algorithm.

[0091] In the embodiments of the present disclosure, after the data to be migrated is migrated to the target database, data verification can also be performed to ensure the accuracy and consistency of the data. Exemplarily, different methods can be used to perform data verification, such as using the LCS algorithm combined with the DIFF tool to perform data verification, or using the Rabin-Karp algorithm and the LCS algorithm to perform data verification.

[0092] S104, when the data verification passes, the migration of the data to be migrated is completed.

[0093] In the embodiments of the present disclosure, if the data verification passes, the migration of the data to be migrated can be considered completed. Exemplarily, the verification can pass, for example, when the data consistency or accuracy meets certain conditions. The certain conditions can be, for example, the number of difference data between the data to be migrated before the migration and the data to be migrated after the migration is less than a preset value, or the proportion of the difference data is less than a preset proportion. Only when the certain conditions are met can the data verification be considered passed and the data migration is completed. In this way, the consistency and accuracy of the data migration can be determined, and the migration quality can be improved.

[0094] It is understandable that if the data verification fails, it is necessary to delete the data migrated this time and re-perform the data migration process, that is, re-execute the method of this embodiment.

[0095] In the disclosed embodiment, a target database is constructed based on the metadata of a source database; the data to be migrated in the source database is migrated to the target database; a preset verification algorithm is used to perform data verification on the data to be migrated; wherein the preset verification algorithm includes a first verification algorithm and a second verification algorithm, the first verification algorithm includes an LCS algorithm and a DIFF tool, and the second verification algorithm includes a Rabin-Karp algorithm and the LCS algorithm; if the data verification passes, the migration of the data to be migrated is completed. In this way, on the one hand, database construction through metadata can simplify the complex process of cross-database migration and achieve efficient migration of database structure, thereby reducing the workload and error rate of manual programming, improving data migration efficiency, and reducing maintenance costs; on the other hand, data consistency verification using the LCS algorithm and the DIFF tool, or the Rabin-Karp algorithm and the LCS algorithm can ensure data consistency and accuracy, thereby improving the efficiency and accuracy of the data migration method.

[0096] In some possible implementations, a preset verification algorithm is used to perform data verification on the data to be migrated, including:

[0097] When the data to be migrated meets the preset conditions, the LCS algorithm and the DIFF tool are used to perform consistency check on the data to be migrated; wherein the preset conditions include at least one of the following: the amount of the data to be migrated is less than the preset amount of data, and the data to be migrated includes a character string;

[0098] When the data to be migrated does not meet the preset conditions, the Rabin-Karp algorithm and the LCS algorithm are used to perform consistency check on the data to be migrated.

[0099] In the embodiments of the present disclosure, different verification algorithms can be selected to perform data verification according to the specific circumstances of the data to be migrated. Exemplarily, it can be determined whether the data to be migrated meets the preset conditions, such as whether the data volume of the data to be migrated is less than the preset data volume. If the data to be migrated meets the preset conditions, such as the data volume of the data to be migrated is less than the preset data volume, the LCS algorithm and the DIFF tool can be used to perform consistency verification on the data to be migrated. Conversely, if the data to be migrated does not meet the preset conditions, such as the data volume of the data to be migrated is greater than or equal to the preset data volume, and the data to be migrated contains a character string, the Rabin-Karp algorithm and the LCS algorithm can be used to perform consistency verification on the data to be migrated.

[0100] In a further possible implementation, the LCS algorithm and the DIFF tool are used to perform consistency check on the data to be migrated, including:

[0101] The longest common subsequence between the data to be migrated and the migrated data is identified by the LCS algorithm; wherein the migrated data is the data obtained after the data to be migrated in the source database is migrated to the target database;

[0102] The longest common subsequence is checked for consistency using the DIFF tool to obtain the check result.

[0103] In the disclosed embodiment, when the LCS algorithm and the DIFF tool are used to perform consistency check on the data to be migrated, the LCS algorithm can be used to first identify the longest common subsequence between the data to be migrated and the data that has been migrated. Then, the DIFF tool is used to perform consistency check on the longest common subsequence between the data to be migrated and the data that has been migrated to obtain a difference result. The difference result can be in the form of a difference report, for example. The difference report can be used to indicate the degree of consistency between the data to be migrated and the data that has been migrated. In this way, by adding the LCS algorithm to the DIFF tool, the amount of calculation can be significantly reduced, the accuracy of problem location can be improved, and complex data types can be supported, thereby realizing timely and accurate discovery and resolution of migration data differences, as well as preventive resolution of migration data difference problems.

[0104] In a further possible implementation, the Rabin-Karp algorithm and the LCS algorithm are used to perform consistency check on the data to be migrated, including:

[0105] Use the Rabin-Karp algorithm to filter the data to be migrated and locate the strings containing specific patterns;

[0106] The LCS algorithm is used to perform consistency check on the character string of a specific pattern to obtain a check result.

[0107] In the disclosed embodiment, when the Rabin-Karp algorithm and the LCS algorithm are used to perform consistency check on the migrated data, the Rabin-Karp algorithm can be used to screen the migrated data first, and quickly locate the strings that may contain specific patterns. Exemplarily, the hash value of each substring in the data set can be quickly calculated by the hash function unique to the Rabin-Karp algorithm, so that the strings with the same hash value can be efficiently selected, thereby effectively reducing the number of subsequent detailed comparisons and improving the overall processing efficiency. Then, for the selected strings, the LCS algorithm can be used for more accurate comparison to find the longest common subsequence between them, so as to perform consistency check, analyze the consistency of the data, and obtain the verification result. Exemplarily, in the precise comparison stage, the LCS algorithm is further used to calculate the longest common subsequence between them, and the consistency of the data can be accurately analyzed by the length of the LCS. In this way, the combined use of the Rabin-Karp algorithm and the LCS algorithm can greatly improve the efficiency and accuracy of the migration of TB-level heterogeneous data warehouses, achieve non-perceptual migration without stopping business, effectively improve the migration efficiency and reduce costs.

[0108] In order to make the data warehouse migration method provided by the embodiment of the present disclosure clearer, the following specific examples are used to illustrate the results. Figure 2 The data warehouse migration method provided by the embodiment of the present disclosure includes the following processing:

[0109] 5.1 Rapidly generate heterogeneous data warehouse models and codes based on metadata to achieve structural migration, that is, heterogeneous data warehouse structure migration based on metadata.

[0110] When performing data migration across data warehouses, the code generation and migration process is complex and time-consuming due to differences in database structure, syntax, and functions. The disclosed embodiment utilizes a metadata-based heterogeneous data warehouse model and code rapid generation technology to complete structure migration and improve data migration efficiency.

[0111] Before the heterogeneous data warehouse migration, recreate the data model in the target database, use the metadata of the source database, parse the DDL code of the source database, convert the code into the syntax of the target database, and then recreate the structural objects in the target database. The following is an example:

[0112] 1. Metadata collection of source database, including metadata collection and standardization: Use the API or tools provided by the database management system (DBMS) to collect metadata of the target database, including table structure, column definition, index, stored procedure, trigger, etc. Convert the collected metadata into a unified target database model format.

[0113] 2. Metadata parsing of the source database: Use predefined parsing rules to parse the standardized metadata and extract key information required for code generation, such as table names, column names, data types, constraints, etc.

[0114] In the data warehouse localization process involved in this embodiment, the parsing rules involved are mainly used to guide how to extract information from Oracle metadata and convert it into GBase metadata. These rules include:

[0115] Data type mapping rules: define a data type mapping table to list Oracle data types and their corresponding GBase data types. During the conversion process, data types are converted according to this mapping table.

[0116] Constraint conversion rules: Determine how to convert constraints in Oracle (such as primary keys, foreign keys) to corresponding constraints in GBase.

[0117] Syntax adjustment rules: For stored procedures and triggers, write rules to convert SQL syntax and function calls to meet the requirements of GBase.

[0118] By setting the above parsing rules, Oracle metadata can be converted into GBase metadata, so as to perform subsequent data conversion, data verification and data cutover.

[0119] 3. Target database structure generation and code generation: Design code templates for different data warehouses based on business needs and technical requirements. Combine the parsed metadata with the corresponding code templates to generate code compatible with the target database system. The generated code includes data access layer (DAL) code, stored procedures, triggers, etc.

[0120] 1) Get table metadata: First, define the get_table_metadata function, input the database type and connection information, and return a list containing table metadata. The information of each table is represented in the form of metadata, including table name, column name, data type, length, and primary key information.

[0121] 2) Define SQL code templates: Define corresponding SQL code templates for different data warehouse systems (this proposal mainly involves Oracle and gbase data warehouses). These templates define how to generate SQL statements for database objects based on metadata, such as SQL statements for creating tables, data access layer (DAL) codes, etc.

[0122] 3) Generate SQL code: define the generate_sql_from_metadata function, input the metadata list and the target database type, select the corresponding SQL code template according to the target database type, traverse each table in the metadata list, fill the table metadata into the template, and generate specific SQL statements.

[0123] 4) Extension to other database objects: Generate code statements related to other database objects, mainly related to stored procedures, triggers, etc. By defining more code templates, referencing different aspects of metadata (such as the logic of stored procedures, trigger conditions, etc.) to generate specific code statements.

[0124] Through the above automated process, the efficiency of database design and deployment is significantly improved, human errors are reduced, and the accuracy and consistency of database objects are ensured.

[0125] 4. Code verification and testing: Verify and test the generated code to ensure its compatibility and correctness with the target database system.

[0126] 5. Code deployment and maintenance: Deploy the verified code to the target environment and establish a corresponding maintenance mechanism so that the code can be regenerated when the database structure changes.

[0127] The above steps can realize the structural migration of the data warehouse. Specifically, the target database model and code automatic generation technology based on metadata greatly reduce the workload of manually writing and modifying database code, reduce the human error rate, support multiple heterogeneous database systems, and achieve cross-platform compatibility. When the database structure changes, the code can be quickly regenerated, which reduces maintenance costs and greatly reduces the code development time for heterogeneous data warehouse migration.

[0128] 5.2 Use triggers to achieve lossless data migration, including lossless migration of full data and incremental data.

[0129] In the related technologies, in the full and incremental data warehouse data migration technologies, there are problems such as lack of unified logs to save data changes, lack of triggers to obtain real-time data change information, delayed data consistency comparison, difficulty in locating data problems and requiring manual intervention and verification, which will affect the consistency of data and data migration efficiency during the migration process.

[0130] This embodiment uses multi-batch real-time incremental data migration based on the traditional full and incremental data migration. It uses log records and triggers to obtain real-time data changes and perform real-time incremental data migration. In addition, data changes are saved in real time to the log table, converted into XML documents, and migrated to the target data warehouse. In this way, when data inconsistencies occur, data problems can be automatically located, reducing manual intervention and achieving automatic lossless data migration. Figure 3 , the specific steps are as follows:

[0131] 1. Real-time capture and logging of data changes:

[0132] In order to obtain data change information in real time and realize real-time incremental data migration, triggers can be created on the synchronization table. The table to be migrated automatically executes the trigger when performing operations such as INSERT, UPDATE, or DELETE. The trigger captures the data changes in the source database in real time, and both the old data and the new data are correctly recorded (for example, saved through log records). These changes are synchronized to the target database in real time, thereby realizing real-time automatic lossless data migration.

[0133] 2. Convert the log to XML document and make a backup:

[0134] Write a stored procedure or PL / SQL block to query the log table and convert the query results into XML format.

[0135] Create a target database or target table structure, and use backup tools such as Data Pump or RMAN (Recovery Manager) to back up the XML documents (or tables containing XML) in the source database to the target database. You can also write PL / SQL code to copy the XML documents to the target database through a database link (DBLINK).

[0136] By effectively preserving log information, when data in the source data warehouse and the target data warehouse are inconsistent, the data problem can be automatically located based on the log information, reducing manual intervention and achieving automatic lossless migration of data.

[0137] 3. Implement full and real-time incremental data transmission, combined with Figure 4 , including the following processing:

[0138] Use Java stored procedures (through JDBC) or external Java applications to access and read XML documents, and send the read XML documents as messages to the OpenJMS message queue.

[0139] At the other end of the message queue, there are one or more consumer applications that listen to the queue in real time and process the received XML messages. These consumer applications parse the XML messages back into the database table format (for example, including XML log parsing, XML log filtering, XML log assembly, XML log synchronization, etc.), and load them into the target database or perform other processing. This enables real-time lossless migration of incremental data and data migration of data warehouses.

[0140] 5.3 Use the DIFF tool optimized by the LCS algorithm to check data consistency.

[0141] The migration of data warehouses needs to ensure data accuracy and reliability. In a heterogeneous data warehouse environment, data consistency checks become very complicated due to differences in data structure, data type, storage method, etc. between different data warehouses. Using the Diff tool to compare the data in two heterogeneous data warehouses, find out the differences between them, and determine whether these differences affect data consistency, can greatly improve the efficiency and accuracy of data migration.

[0142] During the data warehouse migration process, it is necessary to ensure that the data between the source data warehouse and the target data warehouse is consistent, monitor and maintain the data synchronization status between multiple heterogeneous data warehouses, track the changes of data sets over time, and identify incremental data that is added, modified, or deleted, including the following processing:

[0143] 1. Prepare the data warehouse to be relocated: Identify the two heterogeneous data warehouses to be compared, select a diff tool suitable for heterogeneous data warehouses, prepare the comparison environment, and ensure that both data warehouses are accessible.

[0144] 2. Data mapping and conversion: Since there may be differences in data structure and data type between heterogeneous data warehouses, data mapping and conversion are required to ensure the comparability of data.

[0145] 3. Perform Diff comparison: Use the selected Diff tool to perform data comparison and generate a difference report that lists the differences between the two data warehouses in detail.

[0146] 4. Difference analysis and processing: Carefully analyze the difference report to distinguish the differences caused by business logic changes and the errors that may be introduced during the migration process. Based on the analysis results, make necessary repairs or adjustments to the target data warehouse to ensure data consistency.

[0147] As a specific example, combining Figure 5 To check the consistency of user call behavior data in different databases in a heterogeneous data warehouse, you can write a script to extract the data and compare it. Assume that we have two databases: db1 (using MySQL) and db2 (using PostgreSQL). Both databases have a table called user_calls, which contains fields for user call behavior, such as user_id, call_start_time, call_duration, etc. The consistency comparison process can be implemented as follows.

[0148] In actual situations, more complex situations may need to be handled, such as data type conversion, timestamp formatting, data sorting, zipper table processing, etc. For large data sets, it may be impractical to directly compare the entire table, and a more efficient comparison strategy needs to be adopted. This embodiment uses the LCS algorithm based on dynamic rules to improve efficiency.

[0149] The longest common subsequence (LCS) algorithm based on dynamic programming (Dynamic Programming, referred to as dynamic programming) is used to find the length of the longest common subsequence of two (or more) given sequences, and the length is used to determine the consistency of the data. The specific algorithm process is as follows:

[0150] 1. Create a two-dimensional array dp, where dp[i][j] represents the length of the longest common subsequence of sequences X[0..i-1] and Y[0..j-1].

[0151] 2. Initialize the first row and first column of the dp array to 0.

[0152] 3. Traverse each element of X and Y, and for each position (i, j), check the following:

[0153] If X[i-1] is equal to Y[j-1], then dp[i][j] is dp[i-1][j-1]+1.

[0154] If X[i-1] is not equal to Y[j-1], then dp[i][j] is the larger of dp[i-1][j] and dp[i][j-1].

[0155] 4. After the traversal is completed, dp[n][m] contains the length of the longest common subsequence of the two sequences.

[0156] The LCS algorithm adopts the idea of ​​dynamic programming and has high efficiency for comparing large-scale data sets and analyzing the causes of differences. This embodiment adds the LCS algorithm to the DIFF open source code, which significantly improves the efficiency, as follows:

[0157] 1) Reduce the amount of calculation: The LCS algorithm avoids repeated calculation of the same sub-problem through dynamic programming. When comparing two files or data sets, it can find the longest common part between them more quickly and identify differences more efficiently.

[0158] 2) Improve accuracy and precisely locate problems: The LCS algorithm can accurately identify the longest common subsequence between two sequences, which helps the DIFF tool generate difference reports more accurately.

[0159] 3) Support for complex data types: The LCS algorithm is not only applicable to text and string comparisons, but can also be extended to more complex data types, such as codes, database records, etc. Adding the LCS algorithm to the DIFF tool can support more types of data comparisons, thereby discovering and solving problems in advance.

[0160] This embodiment innovatively adds the LCS algorithm to the DIFF tool, which significantly reduces the amount of calculation, improves the accuracy of problem location, and supports complex data types, thereby achieving timely and accurate discovery and resolution of migration data differences, and preventively resolving migration data difference problems.

[0161] 5.4 Using Rabin-Karp algorithm to optimize relocation efficiency

[0162] As mentioned above, for data in the process of migrating large data sets, directly comparing the contents of the entire table or file is very time-consuming and inefficient. In this case, the Rabin-Karp algorithm can be used to improve efficiency.

[0163] The Rabin-Karp algorithm uses hash functions to quickly compare strings and can be used for string matching, such as finding identical substrings in two strings. In the process of migrating TB-level data warehouses, hash technology can be used to check the integrity and consistency of data. During the data extraction phase, the hash value of the data is calculated, and the hash value is calculated again after the data is loaded into the target system for comparison to ensure the accuracy of the data.

[0164] The Rabin-Karp algorithm uses a sliding window, starting from the beginning of the main string and moving rightward character by character. For each substring, the hash value is calculated by the hash function and compared with the hash value of the pattern string. If the hash values ​​are the same, it is further checked whether the substring and the pattern string are completely matched; if the hash values ​​are different, the sliding window is directly moved one character to the right and the matching is continued. The specific steps of the algorithm are as follows: Figure 6 shown.

[0165] The Rabin-Karp algorithm and LCS algorithm are not directly used in the migration process itself, but play an important role in the verification and comparison stages before and after the migration.

[0166] During the migration of a TB-level data warehouse, you can first use the Rabin-Karp algorithm to initially screen a large amount of data and quickly locate strings that may contain specific patterns. Then, for the screened strings, use the LCS algorithm to perform a more accurate comparison to find the longest common subsequence between them, thereby further analyzing the consistency of the data.

[0167] In the initial screening stage of data processing, the hash function unique to the Rabin-Karp algorithm can quickly calculate the hash value of the pattern and each substring in the data set, thereby efficiently selecting strings (substrings) with the same hash value. The time complexity is O(n+m), and the number of subsequent detailed comparisons can be effectively reduced, thereby improving overall processing efficiency.

[0168] In the precise comparison stage, the LCS algorithm is further used to calculate the longest common subsequence between them. The length of LCS can accurately analyze the consistency of the data. The combination of the Rabin-Karp algorithm and the LCS algorithm greatly improves the efficiency and accuracy of the migration of TB-level heterogeneous data warehouses, achieving non-perceptible migration without interrupting business, improving migration efficiency by 33% and reducing costs by 26%.

[0169] It can be seen that, on the one hand, this embodiment introduces metadata technology to quickly generate heterogeneous data warehouse models and codes, collects and analyzes metadata, and combines preset code templates to automatically generate code compatible with the target database, thereby realizing efficient migration of structures and reducing the workload and error rate of manual programming. On the other hand, log records and triggers are used to obtain real-time data changes and perform real-time incremental data migration. The data changes obtained by the trigger are saved in real time to the log table, converted into XML documents, and migrated to the target data warehouse. In this way, when data inconsistencies occur, data problems can be automatically located, manual intervention can be reduced, and real-time automatic lossless migration of data can be realized. At the same time, the LCS algorithm is added to the existing Diff tool. When migrating large data sets, the Diff tool based on the LCS algorithm can reduce the amount of calculation and improve the accuracy of problem location, so as to timely and accurately discover and solve the differences in relocated data. The Diff tool after adding the LCS algorithm can also support complex data types, so that the problem of relocated data differences can be preventively solved. In addition, the Rabin-Karp algorithm can also be used to quickly screen possible matching strings through hash functions, reduce the amount of comparison, and improve the efficiency of preliminary screening. At the same time, combined with the LCS algorithm, the longest common subsequence of the filtered strings is accurately calculated to ensure data consistency and improve the efficiency and accuracy of data migration.

[0170] In summary, the method provided in this embodiment is applicable to the relocation of TB-level heterogeneous data warehouses, data structure migration, full and incremental data migration, data consistency comparison, relocation performance optimization and other fields, and is particularly suitable for heterogeneous data warehouse cutover and data migration after the use of domestic software. Through the method of this embodiment, metadata-driven structure migration technology can be used to simplify cross-database migration, reduce programming workload and error rate; trigger technology can be used to ensure real-time lossless migration of data and maintain data consistency and integrity; DIFF tools and LCS algorithms are used to effectively check data consistency and ensure the reliability of data warehouse migration; the Rabin-Karp algorithm is combined with the LCS algorithm to improve the migration efficiency and accuracy of TB-level data warehouses, and realize non-perceptible relocation without interrupting business. That is, the method provided in this embodiment has at least the following technical effects:

[0171] 1. Metadata-based structural migration technology. The complex process of cross-database migration is simplified through the metadata-driven heterogeneous data warehouse model and code rapid generation technology. By collecting and parsing metadata and combining it with preset code templates, code compatible with the target database is automatically generated to achieve efficient structural migration, reduce the workload and error rate of manual programming, improve data migration efficiency, and reduce maintenance costs.

[0172] 2. Trigger-based real-time incremental data automatic lossless migration technology. Use log records and triggers to obtain real-time data changes and perform real-time incremental data migration. Save the data changes obtained by the trigger to the log table in real time, convert them into XML documents, and migrate them to the target data warehouse. In this way, when data inconsistencies occur, data problems can be automatically located, reducing manual intervention and achieving real-time automatic lossless data migration.

[0173] 3. Data consistency check technology based on the DIFF tool improved by the LCS algorithm. Data warehouse migration needs to ensure data consistency and reliability. This proposal uses the Diff tool to compare the differences between the two data warehouses, identify and process changes that may affect consistency, and correct the target data warehouse through data mapping, conversion, Diff comparison and difference analysis to ensure data consistency. For large data sets, the LCS algorithm based on dynamic programming is used to improve the Diff tool to improve the comparison efficiency and ensure efficient and accurate migration.

[0174] 4. Rapid migration technology based on the Rabin-Karp algorithm. In the migration of TB-level data warehouses, the Rabin-Karp algorithm is used to quickly filter possible matching strings through hash functions, reduce the amount of comparison, and improve the efficiency of preliminary screening. At the same time, combined with the LCS algorithm, the longest common subsequence of the filtered strings is accurately calculated to ensure data consistency, improve the efficiency and accuracy of data migration, and achieve non-perceptible migration without interrupting business, which can effectively improve migration efficiency and reduce migration costs.

[0175] Based on the same inventive concept, the embodiment of the present disclosure also provides a data warehouse migration device. Figure 7 As shown, the data warehouse migration device 700 includes:

[0176] A construction module 710 is used to construct a target database based on the metadata of the source database;

[0177] A migration module 720 is used to migrate the data to be migrated in the source database to the target database;

[0178] A verification module 730, configured to perform data verification on the data to be migrated using a preset verification algorithm; wherein the preset verification algorithm includes a first verification algorithm and a second verification algorithm, the first verification algorithm includes an LCS algorithm and a DIFF tool, and the second verification algorithm includes a Rabin-Karp algorithm and the LCS algorithm;

[0179] The migration module 740 is used to complete the migration of the data to be migrated if the data verification passes.

[0180] In a possible implementation, the construction module 710 is used to:

[0181] Obtain metadata of the source database; wherein the metadata at least includes table structure, column definition, index, stored procedure, and trigger;

[0182] Parsing the metadata by using preset parsing rules to obtain code generation information; wherein the code generation information is used to generate code for a target database, including at least a table name, a column name, a data type, and a constraint condition;

[0183] Generate the code of the target database based on the code generation information and the code template of the preset data warehouse;

[0184] The target database is constructed based on the code of the target database.

[0185] In a possible implementation, the migration module 720 includes:

[0186] A capture unit, used to capture the changes of data in the source database in real time through the trigger to obtain incremental data;

[0187] The migration unit is used to determine the incremental data as the data to be migrated, and migrate the incremental data to the target database.

[0188] In a possible implementation, the migration unit includes:

[0189] A saving subunit, used for saving changes of data acquired through the trigger to a log file;

[0190] A conversion subunit, used for converting the log file into an XML document;

[0191] The migration subunit is used to migrate the XML document to the target database.

[0192] In a possible implementation, the verification module 730 includes:

[0193] A first verification unit, configured to use the LCS algorithm and the DIFF tool to perform consistency verification on the data to be migrated when the data to be migrated meets a preset condition; wherein the preset condition includes at least one of the following: the amount of the data to be migrated is less than a preset amount of data and the data to be migrated includes a character string;

[0194] The second verification unit is used to perform consistency verification on the data to be migrated by using the Rabin-Karp algorithm and the LCS algorithm when the data to be migrated does not meet a preset condition.

[0195] In a possible implementation manner, the first verification unit is used to:

[0196] Identifying the longest common subsequence between the data to be migrated and the migrated data by using the LCS algorithm; wherein the migrated data is data obtained after migrating the data to be migrated in the source database to the target database;

[0197] The DIFF tool is used to perform a consistency check on the longest common subsequence to obtain a check result.

[0198] In a possible implementation manner, the second verification unit is used to:

[0199] Using the Rabin-Karp algorithm to screen the data to be migrated, and locate a character string containing a specific pattern;

[0200] The LCS algorithm is used to perform consistency check on the character string of the specific pattern to obtain a check result.

[0201] The specific implementation method and technical effect of the device provided in the embodiment of the present disclosure are similar to those of the above-mentioned method embodiment, and will not be repeated here.

[0202] According to an embodiment of the present disclosure, the present disclosure also discloses an electronic device, a computer-readable storage medium, and a computer program product.

[0203] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0204] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0205] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0206] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the data warehouse relocation method. For example, in some embodiments, the data warehouse relocation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data warehouse relocation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data warehouse relocation method in any other appropriate manner (e.g., by means of firmware).

[0207] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0208] The program code of the computer program product for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0209] In the context of the present disclosure, a computer-readable storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may be a machine-readable signal medium or a machine-readable storage medium. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0210] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0211] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0212] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0213] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0214] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data warehouse migration method, characterized in that: include: Build the target database based on the metadata of the source database; Migrating the data to be migrated in the source database to the target database; Using a preset verification algorithm to perform data verification on the data to be migrated; wherein the preset verification algorithm includes a first verification algorithm and a second verification algorithm, the first verification algorithm includes an LCS algorithm and a DIFF tool, and the second verification algorithm includes a Rabin-Karp algorithm and the LCS algorithm; When the data verification passes, the migration of the data to be migrated is completed.

2. The data warehouse migration method according to claim 1, characterized in that: The step of constructing a target database based on metadata of a source database includes: Obtain metadata of the source database; wherein the metadata at least includes table structure, column definition, index, stored procedure, and trigger; Parsing the metadata by using preset parsing rules to obtain code generation information; wherein the code generation information is used to generate code for a target database, including at least a table name, a column name, a data type, and a constraint condition; Generate the code of the target database based on the code generation information and the code template of the preset data warehouse; The target database is constructed based on the code of the target database.

3. The data warehouse migration method according to claim 2, characterized in that: The step of migrating the data to be migrated in the source database to the target database includes: Capturing the data changes in the source database in real time through the trigger to obtain incremental data; The incremental data is determined as the data to be migrated, and is migrated to the target database.

4. The data warehouse migration method according to claim 3, characterized in that: The step of determining the incremental data as the data to be migrated and migrating the data to the target database includes: Saving changes to data acquired through the trigger to a log file; Converting the log file into an XML document; Migrate the XML document to the target database.

5. The data warehouse migration method according to claim 1, characterized in that: The using a preset verification algorithm to perform data verification on the data to be migrated includes: In the case where the data to be migrated meets a preset condition, the LCS algorithm and the DIFF tool are used to perform consistency check on the data to be migrated; wherein the preset condition includes at least one of the following: the amount of the data to be migrated is less than a preset amount of data, and the data to be migrated includes a character string; When the data to be migrated does not meet a preset condition, the Rabin-Karp algorithm and the LCS algorithm are used to perform consistency check on the data to be migrated.

6. The data warehouse migration method according to claim 5, characterized in that: The using the LCS algorithm and the DIFF tool to perform consistency check on the data to be migrated includes: Identifying the longest common subsequence between the data to be migrated and the migrated data by using the LCS algorithm; wherein the migrated data is data obtained after migrating the data to be migrated in the source database to the target database; The DIFF tool is used to perform a consistency check on the longest common subsequence to obtain a check result.

7. The data warehouse migration method according to claim 5, characterized in that: The using the Rabin-Karp algorithm and the LCS algorithm to perform consistency check on the data to be migrated includes: Using the Rabin-Karp algorithm to screen the data to be migrated, and locate a character string containing a specific pattern; The LCS algorithm is used to perform consistency check on the character string of the specific pattern to obtain a check result.

8. A data warehouse migration device, characterized in that: include: A construction module, used for constructing a target database based on metadata of a source database; A migration module, used to migrate the data to be migrated in the source database to the target database; A verification module, used to perform data verification on the data to be migrated by using a preset verification algorithm; wherein the preset verification algorithm includes a first verification algorithm and a second verification algorithm, the first verification algorithm includes an LCS algorithm and a DIFF tool, and the second verification algorithm includes a Rabin-Karp algorithm and the LCS algorithm; The migration module is used to complete the migration of the data to be migrated if the data verification passes.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images