Data migration method, device, apparatus, medium, and program product

Abstract

This application provides a data migration method applicable to the big data and financial fields. The method includes: responding to a data migration request, creating a temporary table in a target database corresponding to the target table data, based on the target table data and source table data included in the data migration request; storing a structured partitioned dataset corresponding to the source table data in the temporary table, wherein the partitioned dataset includes a one-to-one correspondence of indexes and data subsets; and migrating the unstructured data subsets in the temporary table to the target partition corresponding to the target table data based on the indexes.

Description

Technical Field

[0001] This application relates to the fields of big data and finance, and specifically to a data migration method, apparatus, device, medium, and program product. Background Technology

[0002] Financial business systems involve a large amount of data processing. However, due to data security considerations, there is no direct network communication interface for data transmission between the databases of two financial business systems, making direct data sharing impossible. Related technologies utilize the respective query engines of the two databases to directly read data from one database and then write the read data to the other database to achieve small-scale data migration.

[0003] In realizing the concept of this application, the related technologies have at least the following problems: the data migration methods in the related technologies require multiple readings and writings of data, and the reading and writing of data will consume a lot of hardware resources, resulting in low data migration efficiency. Summary of the Invention

[0004] In view of the above problems, this application provides data migration methods, apparatus, devices, media and program products to improve data processing efficiency.

[0005] According to a first aspect of this application, a data migration method is provided, comprising: responding to a data migration request, creating a temporary table in a target database corresponding to the target table data, based on target table data and source table data included in the data migration request, wherein the temporary table has the same table structure as the source table corresponding to the source table data; storing a structured partitioned dataset corresponding to the source table data in the temporary table, wherein the partitioned dataset includes a one-to-one corresponding index and a data subset; and migrating an unstructured data subset from the temporary table to the target partition corresponding to the target table data based on the index.

[0006] According to an embodiment of this application, the source table data includes a source database identifier, a source table identifier, and a source partition identifier; the target table data includes a target database identifier; the creation of a temporary table in the target database corresponding to the target table data based on the target table data and source table data included in the data migration request includes: obtaining the table structure definition, partition information, and table metadata information of the source table corresponding to the source table identifier from the source database corresponding to the source database identifier, wherein the table structure definition includes information for creating a blank table with the same structure as the source table; the partition information includes the directory required for querying data corresponding to the source partition identifier; the table metadata information includes multiple log files for recording data changes to the source table; and the creation of the temporary table in the target database corresponding to the target database identifier based on the table structure definition, partition information, and table metadata information of the source table.

[0007] According to an embodiment of this application, the source table data further includes a source partition identifier; storing the structured partition dataset corresponding to the source table data into the temporary table includes: generating a download path based on the source database identifier, the source table identifier, and the source partition identifier; downloading the partition dataset corresponding to the source partition identifier according to the download path; and storing the partition dataset into the temporary table.

[0008] According to an embodiment of this application, when the index is a row identifier, the data subset is the row data corresponding to the row identifier; when the index is a column identifier, the data subset is the column data corresponding to the row identifier.

[0009] According to an embodiment of this application, the target table data further includes a target table identifier and a target partition identifier; the above-mentioned migration of an unstructured data subset from the temporary table to a target partition corresponding to the target table data based on the above-mentioned index includes: reading the data subset based on the above-mentioned index, and writing the data subset into the target partition indicated by the above-mentioned target table identifier and the above-mentioned target partition identifier.

[0010] According to an embodiment of this application, the source database stores multiple data tables, including the source table; the source table includes multiple partitions, including the source partition.

[0011] According to an embodiment of this application, the method further includes, before creating a temporary table in the target database corresponding to the target table data based on the target table data and source table data included in the data migration request: performing compliance verification on the source table data and the target table data to obtain a verification result; if the verification result indicates that the verification is passed, performing the operation of creating the temporary table; after migrating a subset of unstructured data in the temporary table to the target partition corresponding to the target table data based on the index: in response to the completion of the data migration from the source partition corresponding to the source table data to the target partition corresponding to the target table data, deleting the temporary table in the target database.

[0012] A second aspect of this application provides a data migration apparatus, characterized in that the apparatus comprises: a table creation module, configured to, in response to a data migration request, create a temporary table in a target database corresponding to the target table data, based on target table data and source table data included in the data migration request, wherein the temporary table has the same table structure as the source table corresponding to the source table data; a data storage module, configured to store a structured partitioned dataset corresponding to the source table data into the temporary table, wherein the partitioned dataset includes a one-to-one corresponding index and a data subset; and a data migration module, configured to, based on the index, migrate an unstructured data subset from the temporary table to the target partition corresponding to the target table data.

[0013] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0015] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0016] According to an embodiment of this application, a temporary table with the same table structure as the source table is created in the target database. This transforms the data migration process, which originally required multiple input / output operations between databases, into a single data transfer operation. Specifically, the reading and writing processes of data subsets are decoupled. Leveraging the advantages of batch processing, a partitioned dataset is pre-obtained from the source database and stored in the temporary table of the target database, completing the data reading process. Then, based on an index, the data subsets included in the structured partitioned dataset are migrated to the target partition, converting the structured partitioned dataset into an unstructured data subset to meet the storage format requirements of the target partition, completing the data writing process. By sacrificing storage space, the number of data input / output operations between databases is reduced, decreasing the time spent consuming hardware resources of the source database during data migration, thereby improving data migration efficiency. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustration shows an application scenario of the data migration method and apparatus according to embodiments of this application.

[0019] Figure 2 A flowchart illustrating a data migration method according to an embodiment of this application is shown schematically.

[0020] Figure 3 A flowchart illustrating a data migration method according to another embodiment of this application is shown.

[0021] Figure 4A This diagram illustrates the relationship between a database cluster and a source database according to an embodiment of this application.

[0022] Figure 4B The diagram illustrates the relationship between the source database and the source table according to an embodiment of this application.

[0023] Figure 4C The diagram illustrates the relationship between the source database and the source table according to an embodiment of this application.

[0024] Figure 5 An interactive diagram illustrating a data migration method according to an embodiment of this application is shown schematically.

[0025] Figure 6 A schematic diagram of a data migration method according to an embodiment of this application is shown.

[0026] Figure 7A schematic block diagram of a data migration apparatus according to an embodiment of this application is shown.

[0027] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a data migration method according to an embodiment of this application. Detailed Implementation

[0028] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] In the technical solution of this application, all data involved (including but not limited to data used for analysis, data stored, data displayed, etc.) are information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with relevant laws, regulations and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0033] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0034] In the data processing flow involving financial business systems, the current financial business system needs to accurately reuse data from a partition of a data table in another database without overwriting the original data. That is, it needs to obtain data from the database corresponding to the other financial business system and store the obtained data in the database corresponding to the current financial business system.

[0035] Because there are multiple security isolation and compliance barriers between the databases of different financial business systems, data cannot be transmitted directly through the data transmission interface. Therefore, related technologies usually require writing code to read the required data from the database and write the read data into another database used to store the data, so as to facilitate subsequent processing of the read data.

[0036] In actual production, data migration methods in related technologies require calling the query engines of two databases, one for reading and the other for writing data. If the two databases have different data storage formats, and the data to be migrated cannot be completed in a single read and write operation, it will consume the processor and I / O resources of both the source and target databases, impacting the performance of the source database. Especially in high-concurrency scenarios, the query engine may become a performance bottleneck. For example, when multiple data migration tasks run simultaneously, the query engine needs to process multiple query requests concurrently, which may lead to resource contention and decreased database performance.

[0037] Based on this, embodiments of this application provide a data migration method. In response to a data migration request, a temporary table is created in the target database corresponding to the target table data, based on the target table data and source table data included in the data migration request. The temporary table has the same table structure as the source table corresponding to the source table data. The structured partition dataset corresponding to the source table data is stored in the temporary table. The partition dataset includes a one-to-one corresponding index and a data subset. Based on the index, the unstructured data subset in the temporary table is migrated to the target partition corresponding to the target table data.

[0038] Figure 1The illustration shows an application scenario of the data migration method and apparatus according to embodiments of this application.

[0039] like Figure 1 As shown, application scenario 100 according to this embodiment may include a source database 101, a server 102, and a target database 103. A communication link medium exists between the source database 101 and the server 102, and between the target database 103 and the server 102, for communication. The communication link medium may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0040] Server 102 can interact with source database 101 and target database 103 to receive or send instructions or data. The server can have various communication client applications installed, such as web browser applications, search applications, instant messaging tools, etc. (for example only).

[0041] Source database 101 and target database 103 can be databases that provide data storage and management services for different financial business systems. Source database 101 and target database 103 can be two databases located in different database clusters. Target database 103 represents the database from which data is migrated, and source database 101 represents the database from which data is migrated. Server 102 can be a server that provides data management services between various databases. Server 102 can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the data migration method provided in this application embodiment can generally be executed by server 102. Correspondingly, the data migration apparatus provided in this application embodiment can generally be located in server 102. The data migration method provided in this application embodiment can also be executed by a server or server cluster that is different from server 102 and capable of communicating with source database 101 and target database 103 and / or server 102. Correspondingly, the data migration apparatus provided in this application embodiment can also be located in a server or server cluster that is different from server 102 and capable of communicating with source database 101 and target database 103 and / or server 102 and / or server 102.

[0043] It should be understood that Figure 1 The number of databases and servers shown is merely illustrative. Depending on implementation needs, any number of databases and servers can be used.

[0044] The following will be based on Figure 1 The described scene, through Figures 2-6 The data migration method according to the embodiments of this application will be described in detail.

[0045] Figure 2 A flowchart illustrating a data migration method according to an embodiment of this application is shown schematically.

[0046] like Figure 2 As shown, the data migration method of this embodiment includes operations S210 to S230.

[0047] In operation S210, in response to a data migration request, a temporary table is created in the target database corresponding to the target table data, based on the target table data and source table data included in the data migration request. The temporary table has the same table structure as the source table corresponding to the source table data.

[0048] In operation S220, the structured partitioned dataset corresponding to the source table data is stored in a temporary table. The partitioned dataset includes a one-to-one corresponding index and a subset of data.

[0049] In operation S230, based on the index, a subset of unstructured data in the temporary table is migrated to the target partition corresponding to the data in the target table.

[0050] According to embodiments of this application, there may be differences in storage format and data management mechanism between the source database and the target database for data migration. Specifically, the source database is built based on a transactional data lake storage format, and its table structure is transactional, supporting complex data operations and transaction management. The target database builds tables based on file format metadata mapping, and its table structure is defined through file format and metadata mapping.

[0051] In this scenario, the structured partitioned dataset in the source database is organized according to the table structure of the source tables. Without this table structure, the specific content and meaning of the data in the partitioned dataset cannot be determined. Therefore, special attention must be paid to handling this difference during the migration process to ensure data integrity and understandability. This can be understood as the text content of a single page in a book, with the table structure serving as the table of contents and page numbers. Without the table structure, even if the partitioned dataset is transferred to a separate storage space, the relationship between this single page and the entire book cannot be determined, meaning the specific meaning of the data in the partitioned dataset remains unclear.

[0052] Source table data indicates the storage location of the data to be migrated, and may include the storage address of the migrated data in the source database; target table data indicates the location where the data is migrated to, and may include the storage address of the migrated data in the target database.

[0053] The data migration request includes source table data and target table data. Based on the source table data, the table structure of the source table is determined, and a temporary table with the same structure as the source table is created in the target database. A structured partitioned dataset represents a dataset that matches the table structure of the source table. The structured partitioned dataset is obtained from the source table and stored in a temporary table with the same table structure as the source table.

[0054] A structured partitioned dataset can contain multiple data subsets, each with its own index, which identifies the subset. Unstructured datasets, on the other hand, require an index to determine the relative order of the subsets. This can be understood as follows: a partitioned dataset is like the text content of a page in a book, the index is like an identifier dividing that page into segments, and the data subsets are like the corresponding segments. There is a one-to-one correspondence between the index and the data subsets.

[0055] A target partition is a partition stored under a specific table in the target database. During data migration, a temporary table is used to store the structured partition dataset extracted from the source database. These datasets maintain the same structure as the source table in the temporary table. This is done to migrate this data to the target partition in the target database and make it conform to the table structure of the target table containing the target partition.

[0056] Access temporary tables and, based on the indexes in these tables, quickly locate and identify each data subset. For each data subset, convert it to a format suitable for the target partition according to the target database's storage format and metadata mapping requirements, thereby migrating each unstructured data subset to the corresponding target partition.

[0057] According to an embodiment of this application, a temporary table with the same table structure as the source table is created in the target database. This transforms the data migration process, which originally required multiple input / output operations between databases, into a single data transfer operation. Specifically, the reading and writing processes of data subsets are decoupled. Leveraging the advantages of batch processing, a partitioned dataset is pre-obtained from the source database and stored in the temporary table of the target database, completing the data reading process. Then, based on an index, the data subsets included in the structured partitioned dataset are migrated to the target partition, converting the structured partitioned dataset into an unstructured data subset to meet the storage format requirements of the target partition, completing the data writing process. By sacrificing storage space, the number of data input / output operations between databases is reduced, decreasing the time spent consuming hardware resources of the source database during data migration, thereby improving data migration efficiency.

[0058] According to an embodiment of this application, the data migration method further includes, before creating a temporary table in the target database corresponding to the target table data based on the target table data and source table data included in the data migration request: performing compliance verification on the source table data and target table data to obtain a verification result; if the verification result indicates that the verification is passed, performing the operation of creating a temporary table; after migrating a subset of unstructured data in the temporary table to the target partition corresponding to the target table data based on an index: in response to the completion of the data migration from the source partition corresponding to the source table data to the target partition corresponding to the target table data, deleting the temporary table in the target database.

[0059] According to embodiments of this application, compliance verification of source table data and target table data includes: verifying the data format and content of source table data and target table data.

[0060] Validate the format of the source and target table data. For example, verify whether the source and target table data conform to a lightweight data exchange format.

[0061] Validate the content of the source table data and the target table data. For example, check whether the source table data and the target table data contain missing fields.

[0062] The source table data may include the source database identifier, the source table identifier, and the source partition identifier. The target table data may include the target database identifier, the target table identifier, and the target partition identifier. The contents of the source table data and the target table data are validated to determine whether any of the following are missing: source database identifier, source table identifier, source partition identifier, target database identifier, target table identifier, and target partition identifier.

[0063] According to the embodiments of this application, compliance verification is used to determine in advance whether the format and content of the source table data and the target table data meet the migration requirements. Due to problems such as missing content or mismatched format of the source table data and the target table data, the operations that have already been executed become invalid, so as to reduce invalid operations and reduce the occupation of database hardware resources.

[0064] If the verification result indicates that the verification is successful, a temporary table is created to migrate multiple data subsets from the partitioned dataset to the target partition. After migrating the unstructured data subsets from the temporary table to the target partition corresponding to the target table data based on the index: in response to the completion of the data migration from the source partition corresponding to the source table data to the target partition corresponding to the target table data, the temporary table in the target database is deleted, thereby reducing the consumption of storage resources in the target database.

[0065] If the verification result indicates that the verification fails, a verification failure message is generated to indicate that there is an error in the information in the data migration request, so that staff can promptly identify the reason why the data migration cannot be carried out.

[0066] Figure 3 A flowchart illustrating a data migration method according to another embodiment of this application is shown.

[0067] like Figure 3 As shown, the data migration method of this embodiment includes operations S310 to S340.

[0068] When operating S310, the configuration information of the source table is obtained from the source database based on the source database identifier and the source table identifier.

[0069] The configuration information for the source table includes the table structure definition, partition information, and table metadata information.

[0070] When operating the S320, a temporary table is created in the target database based on the configuration information.

[0071] The temporary table is a mirror table of the source table that does not store data.

[0072] During operation S330, a structured partition dataset is obtained from the source database based on the source table database identifier, source table identifier, and source partition identifier, and stored in a temporary table.

[0073] The partitioned dataset includes a one-to-one correspondence between indexes and data subsets.

[0074] In operation S340, based on the index, a subset of unstructured data from a temporary table is migrated to the target partition corresponding to the data in the target table.

[0075] According to an embodiment of this application, the source table data includes a source database identifier, a source table identifier, and a source partition identifier; the target table data includes a target database identifier; based on the target table data and source table data included in the data migration request, a temporary table is created in the target database corresponding to the target table data, including: obtaining the table structure definition, partition information, and table metadata information of the source table corresponding to the source table identifier from the source database corresponding to the source database identifier, wherein the table structure definition includes information for creating a blank table with the same structure as the source table; the partition information includes the directory required to query the data corresponding to the source partition identifier; the table metadata information includes multiple log files for recording data changes in the source table; and a temporary table is created in the target database corresponding to the target database identifier based on the table structure definition, partition information, and table metadata information of the source table.

[0076] According to embodiments of this application, the source table data includes a source database identifier. When the source database is any database in a database cluster, it can be accessed through the source database identifier. The source database stores the table structure definition, partition information, and table metadata information of the source table. These information are typically stored in the root directory of the source table. The root directory of the source table can be accessed through the source table identifier to obtain the table structure definition, partition information, and table metadata.

[0077] The table structure definition specifies the multiple fields included in the source table, as well as the data types of each field. A field can be an identifier for a column or row of data in the source table, used to represent the attribute of that column or row. The data type is used to constrain the value format of the data in that column; for example, data types can include integers, characters, strings, etc.

[0078] As can be understood, a field is the basic data storage unit of a source table, used to store specific types of data. The source partition, corresponding to the source partition identifier, is the logical division of data in the source table, used to improve the efficiency of data management and querying. Fields are used to store data, while source partitions are used to organize and manage data.

[0079] For example, if the source table includes the financial product identifier, financial product name, and financial product sales date of multiple financial products, then the fields of the source table include the financial product identifier, financial product name, and financial product sales date. The source partition can include the financial product identifier, financial product name, and financial product sales date of all financial products with the financial product name A.

[0080] Table metadata includes multiple log files that record data changes to the source table. Specifically, these log files record the operation type, timestamp, and operation object during the data change process. The operation type includes write, delete, and rollback. Table metadata may also include configuration information such as the table type of the source table.

[0081] Partition information includes the directory required to query data corresponding to the source partition identifier, which records which partitions are included in the source table and the identifier of each partition.

[0082] Based on the table structure definition, partition information, and table metadata of the source table, a mirror temporary table that is completely identical to the source table is created in the target database so that the partitioned dataset can still be accessed after being saved to the temporary table.

[0083] According to the embodiments of this application, the table structure definition, partition information and table metadata information of the source table are obtained from the source database, and a temporary table is created in the target database so that the temporary table has the same structure as the source table, which facilitates data migration and subsequent operations.

[0084] According to an embodiment of this application, the source table data further includes a source partition identifier; storing the structured partition dataset corresponding to the source table data into a temporary table includes: generating a download path based on the source database identifier, the source table identifier, and the source partition identifier; downloading the partition dataset corresponding to the source partition identifier based on the download path; and storing the partition dataset into a temporary table.

[0085] The source database identifier is used to identify the database in the database cluster, the source table identifier is used to identify the source table among the multiple data tables included in the source database, and the source partition identifier can be used to identify the source partition among multiple partitions.

[0086] According to an embodiment of this application, the source database stores multiple data tables, including source tables; the source tables include multiple partitions, including source partitions.

[0087] The source or target database can be a database belonging to different database clusters, as will be discussed below. Figures 4A-4C This section introduces the relationships between database clusters, source databases, source tables, and source partitions.

[0088] Figure 4A This diagram illustrates the relationship between a database cluster and a source database according to an embodiment of this application.

[0089] like Figure 4A As shown, the database cluster includes multiple databases, namely database 400_1 to database 400_n, where n is a positive integer greater than 1. Assume database 400_2 is the source database.

[0090] Figure 4B The diagram illustrates the relationship between the source database and the source table according to an embodiment of this application.

[0091] like Figure 4B As shown, assuming database 400_2 is the source database, the source database 400_2 includes multiple data tables, namely data table 400_2_1 to data table 400_2_m, where m is a positive integer greater than 1.

[0092] Figure 4C The diagram illustrates the relationship between the source database and the source table according to an embodiment of this application.

[0093] like Figure 4C As shown, assuming the source table is 400_2_3, which includes multiple data partitions, specifically source partition 401. The download path is typically a file system path or a Uniform Resource Locator (URL) used to locate and access data. Combining the source database identifier, source table identifier, and source partition identifier generates a complete download path pointing to the source partition dataset.

[0094] Using the generated download path, download the partition dataset corresponding to the source partition identifier from the source database. This step ensures data integrity and consistency. Store the downloaded partition dataset in a temporary table in the target database. The temporary table has the same structure as the source table to ensure data consistency during migration.

[0095] According to embodiments of this application, the required partition dataset can be accurately located by using the source database identifier, source table identifier, and source partition identifier, avoiding unnecessary data scanning and reading operations and improving the efficiency of data retrieval.

[0096] According to an embodiment of this application, the source database can be located in a database cluster. When accessing a partitioned dataset of the source database, the login information of the visitor can be cached in a certain location. The cached login information is called cached credentials. The location of the credential cache can be set to a temporary file. The filename of this temporary file contains the environment name of the development environment, test environment, or production environment. This allows for setting an independent credential cache for each environment, preventing interference between different environments, improving the stability and security of the database, and reducing data loss.

[0097] According to an embodiment of this application, the target table data further includes a target table identifier and a target partition identifier; based on the index, migrating a subset of unstructured data in the temporary table to the target partition corresponding to the target table data includes: reading the data subset based on the index, so as to write the data subset into the target partition indicated by the target table identifier and the target partition identifier.

[0098] According to an embodiment of this application, it is necessary to write data from multiple data subsets to a target partition. The target database stores multiple data tables, including a target table. Each target table includes multiple partitions, and each partition includes the target partition. The write location of the data subsets can be determined based on the target table identifier and the target partition identifier.

[0099] A partitioned dataset can include multiple subsets of data, with each subset corresponding to an index. The subsets are partitioned based on the fields of the table, and the index can be a table field.

[0100] Based on the index, data subsets are read sequentially from the temporary table. The write location of the data subset is determined according to the target table identifier and the target partition identifier. Field mapping is performed on the target partition according to the index so that the fields corresponding to the data subsets are consistent with the field structure of the target table, so as to write multiple data subsets to the corresponding locations of the target partitions.

[0101] According to an embodiment of this application, when the index is a row identifier, the data subset is the row data corresponding to the row identifier; when the index is a column identifier, the data subset is the column data corresponding to the row identifier.

[0102] According to an embodiment of this application, when the index is a row identifier, that is, when the data subset is stored row by row, row data is read based on the index to obtain the data subset corresponding to the index.

[0103] When the index is used as a column identifier, that is, when the data subset is stored column-wise, column data is read based on the index to obtain the data subset corresponding to the index.

[0104] According to embodiments of this application, a subset of data is read based on an index, and the structured partitioned dataset is stored in the target partition in an unstructured form to adapt to the target partition's storage format and management mechanism. This approach ensures that data can be efficiently stored, managed, and queried in the target environment, while providing flexibility for subsequent data processing and analysis.

[0105] Figure 5 An interactive diagram illustrating a data migration method according to an embodiment of this application is shown schematically.

[0106] like Figure 5 As shown, the interaction process between the source database, the server, and the target database in this embodiment includes operations S501 to S508.

[0107] When operating S501, send an information retrieval command.

[0108] In response to a user's data migration request, the server sends an information retrieval instruction to the source database based on the source database identifier and source table identifier included in the data migration request. This instruction is used to retrieve the table structure definition, partition information, and table metadata information of the source table corresponding to the source table identifier.

[0109] Operation S502 returns the table structure definition, partition information, and table metadata information of the source table.

[0110] The source database returns the table structure definition, partition information, and table metadata information of the source table corresponding to the source table identifier.

[0111] In operation S503, a temporary table creation command is sent.

[0112] The server sends instructions to the target database to create a temporary table based on the table structure definition, partition information, and table metadata information of the source table, in order to create a temporary table in the target database corresponding to the target database identifier.

[0113] When operating S504, a download path is generated.

[0114] The server generates the download path based on the source database identifier, source table identifier, and source partition identifier.

[0115] When operating the S505, a data download command is sent.

[0116] The server sends a data download instruction to the source database so that the partitioned dataset can be retrieved from the source database according to the download path.

[0117] When operating S506, receive partitioned datasets.

[0118] The server receives the partitioned dataset returned by the source database.

[0119] In operation S507, the partitioned dataset is stored in a temporary table.

[0120] The server stores the partitioned dataset in a temporary table.

[0121] When operating S508, send data migration instructions.

[0122] The server sends a data migration instruction to the target database to determine the target database's query engine based on the target database identifier and then invokes that engine. Since the temporary table is created based on the source table, its identifier is the same as the source table identifier. Therefore, during the query engine invocation process, the temporary table can be accessed in the target database based on the source table identifier, and the unstructured subset of data in the temporary table can be migrated to the target partition corresponding to the target table data based on the index.

[0123] Figure 6 A schematic diagram of a data migration method according to an embodiment of this application is shown.

[0124] like Figure 6 As shown, based on the target table data and source table data included in the data migration request, a temporary table 600_3_3 is created in the target database 600_3 corresponding to the target table data. The temporary table 600_3_3 has the same table structure as the source table 400_2_3 corresponding to the source table data, facilitating the copying of the partition dataset of source partition 401 included in the source table in source database 400_2 to the temporary table 600_3_3. The partition dataset includes a one-to-one correspondence of indexes and data subsets. Based on the indexes, the unstructured data subset in the temporary table is migrated to the target partition 601 corresponding to the target table data 600_3_1. Based on the above data migration method, this application also provides a data...

[0125] Migration device. The following will combine... Figure 7 The device is described in detail.

[0126] Figure 7 A schematic block diagram of a data migration apparatus according to an embodiment of this application is shown.

[0127] like Figure 7As shown, the data migration device 700 of this embodiment includes a table creation module 710, a data storage module 720, and a data migration module 730.

[0128] The table creation module 710 is used to respond to a data migration request and create a temporary table in the target database corresponding to the target table data based on the target table data and source table data included in the data migration request. The temporary table has the same table structure as the source table corresponding to the source table data. In one embodiment, the table creation module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0129] The data storage module 720 is used to store the structured partitioned dataset corresponding to the source table data into a temporary table, wherein the partitioned dataset includes a one-to-one corresponding index and a subset of data; in one embodiment, the data storage module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0130] The data migration module 730 is used to migrate a subset of unstructured data from a temporary table to a target partition corresponding to the data in the target table, based on an index. In one embodiment, the data migration module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0131] According to an embodiment of this application, the source table data includes a source database identifier, a source table identifier, and a source partition identifier; the target table data includes a target database identifier; the table creation module 710 includes: an information acquisition submodule and a table creation submodule.

[0132] The information acquisition submodule is used to retrieve the table structure definition, partition information, and table metadata information of the source table corresponding to the source table identifier from the source database corresponding to the source database identifier. The table structure definition includes information for creating a blank table with the same structure as the source table. The partition information includes the directory required to query the data corresponding to the source partition identifier. The table metadata information includes multiple log files used to record data changes to the source table.

[0133] The table creation submodule is used to create temporary tables in the target database corresponding to the target database identifier, based on the table structure definition, partition information, and table metadata information of the source table.

[0134] According to an embodiment of this application, the source table data further includes a source partition identifier. The data storage module 720 includes: a path generation submodule, a download submodule, and a data storage submodule.

[0135] The path generation submodule is used to generate download paths based on the source database identifier, source table identifier, and source partition identifier.

[0136] The download submodule is used to download the partition dataset corresponding to the source partition identifier based on the download path.

[0137] The data storage submodule is used to store partitioned datasets into temporary tables.

[0138] According to embodiments of this application, when the index is a row identifier, the data subset is the row data corresponding to the row identifier. When the index is a column identifier, the data subset is the column data corresponding to the row identifier.

[0139] According to an embodiment of this application, the target table data further includes a target table identifier and a target partition identifier. The data migration module 730 includes a data writing submodule.

[0140] The data writing submodule is used to read a subset of data based on an index and write the subset of data to the target partition indicated by the target table identifier and the target partition identifier.

[0141] According to an embodiment of this application, the source database stores multiple data tables, including source tables. Each source table includes multiple partitions, including source partitions.

[0142] According to an embodiment of this application, the data migration device 700 further includes a verification module and a deletion module.

[0143] The verification module performs compliance checks on the source and source table data before creating a temporary table in the target database corresponding to the target table data, based on the target table data and source table data included in the data migration request. The verification result is then used to obtain the verification result. If the verification passes, the temporary table creation operation is executed.

[0144] The delete module is used after migrating an unstructured subset of data from a temporary table to a target partition corresponding to the target table data, based on an index: in response to the completion of the data migration from the source partition corresponding to the source table data to the target partition corresponding to the target table data, the temporary table in the target database is deleted.

[0145] According to embodiments of this application, any plurality of modules among the table creation module 710, data storage module 720, and data migration module 730 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the table creation module 710, data storage module 720, and data migration module 730 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the table creation module 710, data storage module 720, and data migration module 730 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0146] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a data migration method according to an embodiment of this application.

[0147] like Figure 8 As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0148] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0149] According to embodiments of this application, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0150] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0151] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0152] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data migration method provided in the embodiments of this application.

[0153] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0155] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0156] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0158] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A data migration method, characterized in that, The method includes: In response to a data migration request, a temporary table is created in the target database corresponding to the target table data, based on the target table data and source table data included in the data migration request. The temporary table has the same table structure as the source table corresponding to the source table data. The structured partitioned dataset corresponding to the source table data is stored in the temporary table, wherein the partitioned dataset includes a one-to-one corresponding index and a subset of data; Based on the index, a subset of unstructured data in the temporary table is migrated to the target partition corresponding to the data in the target table.

2. The method according to claim 1, characterized in that, The source table data includes a source database identifier, a source table identifier, and a source partition identifier; the target table data includes a target database identifier. The step of creating a temporary table in the target database corresponding to the target table data, based on the target table data and source table data included in the data migration request, includes: From the source database corresponding to the source database identifier, obtain the table structure definition, partition information, and table metadata information of the source table corresponding to the source table identifier. The table structure definition includes information for creating a blank table with the same structure as the source table. The partition information includes the directory required to query the data corresponding to the source partition identifier. The table metadata information includes multiple log files for recording data changes to the source table. Based on the table structure definition of the source table, the partition information, and the table metadata information, the temporary table is created in the target database corresponding to the target database identifier.

3. The method according to claim 2, characterized in that, The source table data also includes the source partition identifier; The step of storing the structured partitioned dataset corresponding to the source table data into the temporary table includes: A download path is generated based on the source database identifier, the source table identifier, and the source partition identifier; Download the partition dataset corresponding to the source partition identifier according to the download path; The partitioned dataset is stored in the temporary table.

4. The method according to claim 1, characterized in that, When the index is a row identifier, the data subset is the row data corresponding to the row identifier; when the index is a column identifier, the data subset is the column data corresponding to the row identifier.

5. The method according to claim 2, characterized in that, The target table data also includes the target table identifier and the target partition identifier; The step of migrating a subset of unstructured data from the temporary table to a target partition corresponding to the data in the target table, based on the index, includes: The data subset is read based on the index and written to the target partition indicated by the target table identifier and the target partition identifier.

6. The method according to claim 3, characterized in that, The source database stores multiple data tables, including the source table; the source table includes multiple partitions, including the source partition.

7. The method according to claim 1, characterized in that, The method further includes, before creating a temporary table in the target database corresponding to the target table data based on the target table data and source table data included in the data migration request: The source table data and the target table data are subjected to compliance verification to obtain the verification results; If the verification result indicates that the verification is successful, the operation of creating the temporary table is performed; After migrating the unstructured subset of data from the temporary table to the target partition corresponding to the data in the target table, based on the index: Upon completion of the data migration from the source partition corresponding to the source table data to the target partition corresponding to the target table data, the temporary table in the target database is deleted.

8. A data migration device, characterized in that, The device includes: The table creation module is used to respond to a data migration request and create a temporary table in the target database corresponding to the target table data, based on the target table data and source table data included in the data migration request. The temporary table has the same table structure as the source table corresponding to the source table data. The data storage module is used to store the structured partitioned dataset corresponding to the source table data into the temporary table, wherein the partitioned dataset includes a one-to-one corresponding index and a data subset; The data migration module is used to migrate a subset of unstructured data from the temporary table to a target partition corresponding to the data in the target table, based on the index.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images