A data migration method, device and computer equipment

Abstract

The application relates to a data migration method and device and a computer device. The method comprises the following steps: in response to a data migration instruction, obtaining to-be-migrated data in a to-be-migrated database, generating structured export data and unstructured export data based on an export tool of the to-be-migrated database; analyzing the structured export data based on configuration information of the to-be-migrated database to obtain a query statement of the structured export data, wherein the configuration information comprises a data table name and a field name; analyzing the unstructured export data based on a data structure of the unstructured export data to obtain a query statement of the unstructured export data, wherein the data structure comprises starting data, data length and data address; and splicing the query statement of the structured export data and the query statement of the unstructured export data, and importing the to-be-migrated data into a target database according to the spliced query statement. The method can realize migration of data between different types of databases.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data migration method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] When an application system is modified and the database platform it uses changes, it is necessary to retain the original data. However, different database products have different field type definitions, so it is necessary to migrate the original data to the new database platform.

[0003] In related technologies, manual operation is usually required for each piece of data to be migrated, which greatly increases the workload of staff, wastes human and material resources, and reduces the accuracy of database migration. Summary of the Invention

[0004] Based on this, it is necessary to provide a data migration method to address the aforementioned technical problems. This method can parse the data exported from the source database line by line, process it according to different database types, and then insert the data into the target database in batches, thus realizing the migration of data between different types of databases.

[0005] Firstly, this application provides a data migration method. The method includes:

[0006] In response to the data migration command, the system retrieves the data to be transferred from the database to be transferred, and generates structured export data and unstructured export data based on the export tool of the database to be transferred.

[0007] Based on the configuration information of the database to be transferred, the structured export data is parsed to obtain the query statement for the structured export data. The configuration information includes the data table name and field names.

[0008] The unstructured exported data is parsed based on the data structure of the unstructured exported data to obtain the query statement of the unstructured exported data. The data structure includes the starting data, the data length, and the data address.

[0009] The query statement for the structured exported data is concatenated with the query statement for the unstructured exported data, and the data to be transferred is imported into the target database based on the concatenated query statement.

[0010] In one embodiment, parsing the structured exported data based on the configuration information of the database to be transferred includes:

[0011] Based on the table name and field names in the configuration information, each row of structured exported data in the data table is split into columns to obtain the data structure of each structured exported data. The data structure includes the position, order, and value of the structured exported data.

[0012] In one embodiment, the query statement for obtaining structured exported data includes:

[0013] Based on the table name, field names, and data structure, obtain the query statement for structured data export.

[0014] In one embodiment, the parsing of the unstructured exported data based on the data structure of the unstructured exported data includes:

[0015] The location of the unstructured exported data in the data table is obtained based on the data address, and the corresponding length of unstructured exported data is read from the starting data.

[0016] In one embodiment, the export tool based on the database to be transferred generates structured export data and unstructured export data, including:

[0017] Select the appropriate export tool for structured and unstructured data based on the type of database to be transferred.

[0018] In one embodiment, importing the data to be transferred into the target database according to the concatenated query statement includes:

[0019] The query statement generates a batch data import request to the target database, and the batch data import request is assembled into a single query statement for merging.

[0020] Secondly, this application also provides a data migration apparatus, the apparatus comprising:

[0021] The export module is used to respond to data migration instructions, obtain the data to be transferred from the database to be transferred, and generate structured export data and unstructured export data based on the export tool of the database to be transferred.

[0022] The first parsing module is used to parse the structured exported data based on the configuration information of the database to be transferred, and obtain the query statement of the structured exported data. The configuration information includes the data table name and field name.

[0023] The second parsing module is used to parse the unstructured exported data based on the data structure of the unstructured exported data to obtain the query statement of the unstructured exported data. The data structure includes the starting data, the data length, and the data address.

[0024] The import module is used to concatenate the query statements of the structured exported data with the query statements of the unstructured exported data, and import the data to be transferred into the target database according to the concatenated query statements.

[0025] In one embodiment, parsing the structured exported data based on the configuration information of the database to be transferred includes:

[0026] Based on the table name and field names in the configuration information, each row of structured exported data in the data table is split into columns to obtain the data structure of each structured exported data. The data structure includes the position, order, and value of the structured exported data.

[0027] In one embodiment, the query statement for obtaining structured exported data includes:

[0028] Based on the table name, field names, and data structure, obtain the query statement for structured data export.

[0029] In one embodiment, the parsing of the unstructured exported data based on the data structure of the unstructured exported data includes:

[0030] The location of the unstructured exported data in the data table is obtained based on the data address, and the corresponding length of unstructured exported data is read from the starting data.

[0031] In one embodiment, the export tool based on the database to be transferred generates structured export data and unstructured export data, including:

[0032] Select the appropriate export tool for structured and unstructured data based on the type of database to be transferred.

[0033] In one embodiment, importing the data to be transferred into the target database according to the concatenated query statement includes:

[0034] The query statement generates a batch data import request to the target database, and the batch data import request is assembled into a single query statement for merging.

[0035] Thirdly, this disclosure also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a data migration method.

[0036] Fourthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of a data migration method.

[0037] Fifthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of a data migration method.

[0038] The above data migration methods have at least the following beneficial effects:

[0039] The embodiments provided in this disclosure can parse the data exported from the source database line by line, process it according to different database types, and then insert the data into the target database in batches, thereby realizing the migration of data between different types of databases.

[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments or conventional technologies of this disclosure, the accompanying drawings used in the description of the embodiments or conventional technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a diagram illustrating the application environment of a data migration method in one embodiment.

[0043] Figure 2 This is a flowchart illustrating a data migration method in one embodiment;

[0044] Figure 3 This is a flowchart illustrating a data migration method in one embodiment;

[0045] Figure 4 This is a structural block diagram of a data migration device in one embodiment;

[0046] Figure 5 This is a structural block diagram of a data migration device in one embodiment;

[0047] Figure 6 This is an internal structural diagram of a computer device in one embodiment;

[0048] Figure 7 This is an internal structure diagram of a server in one embodiment. Detailed Implementation

[0049] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0050] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any specific order.

[0051] This disclosure provides a data migration method that can be applied to, for example... Figure 1 In the application environment shown, the database to be transferred 102 sends the data to be transferred to the server 104. The server 104 parses the data to be transferred, obtains data that matches the type of the target database 106, and imports the data into the target database 106.

[0052] In some embodiments of this disclosure, such as Figure 2 As shown, a data migration method is provided, which is applied to... Figure 1 The method is illustrated using the example of a server processing data to be migrated. It is understood that this method can be applied to a server, and also to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In a specific embodiment, the method may include the following steps:

[0053] S202: In response to the data migration instruction, obtain the data to be transferred from the database to be transferred, and generate structured export data and unstructured export data based on the export tool of the database to be transferred.

[0054] Database migration instructions are commands used to migrate data from one database to another. The database to be migrated refers to the database to be migrated, and the target database refers to the database to which the data is to be migrated. In this embodiment, the source and target databases are heterogeneous databases, for example, the source database is an Oracle database and the target database is a MySQL database. Due to the differences in database language syntax, heterogeneous databases require syntax transformation during data migration.

[0055] Structured exported data can include data that can be represented by numbers or a uniform structure, such as numbers and symbols. Unstructured exported data can include data that cannot be represented by numbers or a uniform structure, such as images, sounds, and web pages. In some embodiments of this disclosure, structured exported data can include text data, which can be readable text information, while unstructured exported data can include large object data (Lob), which can be unreadable binary information. The data to be transferred may also include empty fields. If an empty field is encountered, it will be occupied by a subsequent field with data, which will lead to a misalignment of column numbers. To avoid this phenomenon, we can use other symbols to represent empty fields, which can be determined according to the type of the target database.

[0056] Different databases typically use different export tools. For example, for Informix databases, unload is used to export structured and unstructured data, while for MySQL databases, mysqldump is used to export both structured and unstructured data.

[0057] S204: Based on the configuration information of the database to be transferred, the structured export data is parsed to obtain the query statement of the structured export data. The configuration information includes the data table name and field names.

[0058] Configuration information may include database type, table name, field name, column number of unstructured exported data field, data import rules, etc. Data import rules include adding data, deleting data, replacing data, etc.

[0059] A database is a repository for organizing, storing, and managing data according to data structures. Broadly speaking, it's a library for storing data; narrowly defined, it's a structured table stored in a computer. The purpose of a database is to record information. A data table is a temporary grid-like virtual table for storing data, consisting of a table name, fields, and records. Fields can represent variables associated with objects or classes. For example, a two-dimensional table named "Democracy Information" could have fields like name, gender, and age. Based on the table name and field names, the structured data is parsed to obtain the query statement for that structured data.

[0060] S206: Parse the unstructured exported data based on the data structure of the unstructured exported data to obtain the query statement of the unstructured exported data. The data structure includes the starting data, the data length, and the data address.

[0061] If a column of unstructured exported data contains "1db8,b3f,blob5c29.71f", separating this content with commas will result in an array containing three elements: ["1db8", "b3f", "blob5c29.71f"]. The first element, "1db8", represents the starting byte; the second element, "b3f", represents the length of the unstructured exported data; and the third element, "blob5c29.71f", represents the filename containing the unstructured exported data. Based on the data structure of the unstructured exported data, the query statement for the unstructured exported data can be obtained.

[0062] S208: Concatenate the query statement of the structured exported data with the query statement of the unstructured exported data, and import the data to be transferred into the target database according to the concatenated query statement.

[0063] The exported data in the database is parsed row by row to obtain various data types. Different data types are processed accordingly to generate the data required by the target database. The data to be transferred is then inserted into the target database according to the query statement.

[0064] The above data migration method involves parsing the data exported from the source database line by line, processing it according to different database types, and then inserting the data into the target database in batches, thus realizing the migration of data between different types of databases.

[0065] In some embodiments of this disclosure, parsing the structured exported data based on the configuration information of the database to be transferred includes:

[0066] Based on the table name and field names in the configuration information, each row of structured exported data in the data table is split into columns to obtain the data structure of each structured exported data. The data structure includes the position, order, and value of the structured exported data.

[0067] Based on the table name and field names in the configuration information, each row of structured exported data in the data table is split into columns to obtain the data structure of each structured exported data, which can include the position, order, and current value of the data in the data table.

[0068] In some embodiments of this disclosure, the query statement for obtaining structured exported data includes:

[0069] Based on the table name, field names, and data structure, obtain the query statement for structured data export.

[0070] The query statement can be formatted as insert into table_name(field_name1, field_name2, ..., field_namen)values(value1, value2, ..., valuen), where the table name and field names come from the configuration file, and the values ​​come from the structured exported data, thus generating the query statement.

[0071] In some embodiments of this disclosure, the parsing of unstructured derived data based on the data structure of the unstructured derived data includes:

[0072] The location of the unstructured exported data in the data table is obtained based on the data address, and the corresponding length of unstructured exported data is read from the starting data.

[0073] If a column of unstructured exported data contains the following content: "1db8,b3f,blob5c29.71f", separating this content with commas will result in an array containing three elements: ["1db8", "b3f", "blob5c29.71f"]. The first element "1db8" represents the starting byte, the second element "b3f" represents the length of the LOB content, and the third element "blob5c29.71f" represents the filename of the unstructured exported data. Therefore, the file "blob5c29.71f" can be read, starting from "1db8" and reading "b3f" bytes to obtain the content of this unstructured exported data.

[0074] In some embodiments of this disclosure, generating structured export data and unstructured export data based on the export tool of the database to be transferred includes:

[0075] Select the appropriate export tool for structured and unstructured data based on the type of database to be transferred.

[0076] Select the appropriate export tool based on the type of database to be transferred. Further select the appropriate export tool based on whether the data is structured or unstructured. For example, use mysqldump to export both structured and unstructured data for MySQL databases, use Spool to export structured data for Oracle databases, and use UTL to export unstructured data.

[0077] In some embodiments of this disclosure, Figure 3 This is a flowchart illustrating a data migration method in one embodiment.

[0078] The step of importing the data to be transferred into the target database based on the concatenated query statement includes:

[0079] S302: Submit a batch import data request to the target database for the query statement, and perform a merge process on the batch import data request to assemble it into a single query statement.

[0080] The generated query statements can be added to a batch for execution, and executed once every 1000 records in the batch. For tables that were successfully imported, the table name is recorded; for tables that failed to import, the table name and details of the failed data are also recorded.

[0081] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0082] Based on the same inventive concept, this disclosure also provides a data migration apparatus for implementing the data migration method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in the data migration apparatus embodiments provided below can be found in the limitations of the data migration method described above, and will not be repeated here.

[0083] The apparatus may include a system (including a distributed system), software (application), module, component, server, client, etc., that uses the methods described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the apparatuses in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the apparatus are similar, the implementation of the specific apparatus in the embodiments of this specification can refer to the implementation of the foregoing methods, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatuses described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0084] In one embodiment, such as Figure 4As shown, a data migration device 400 is provided. The device can be the aforementioned server, or a module, component, device, or unit integrated into the server. The device 400 may include:

[0085] The export module 402 is used to respond to the data migration instruction, obtain the data to be transferred from the database to be transferred, and generate structured export data and unstructured export data based on the export tool of the database to be transferred.

[0086] The first parsing module 404 is used to parse the structured exported data based on the configuration information of the database to be transferred, and obtain the query statement of the structured exported data. The configuration information includes the data table name and field name.

[0087] The second parsing module 406 is used to parse the unstructured exported data based on the data structure of the unstructured exported data to obtain the query statement of the unstructured exported data. The data structure includes the starting data, the data length, and the data address.

[0088] Import module 408 is used to concatenate the query statement of the structured exported data with the query statement of the unstructured exported data, and import the data to be transferred into the target database according to the concatenated query statement.

[0089] In one embodiment, parsing the structured exported data based on the configuration information of the database to be transferred includes:

[0090] Based on the table name and field names in the configuration information, each row of structured exported data in the data table is split into columns to obtain the data structure of each structured exported data. The data structure includes the position, order, and value of the structured exported data.

[0091] In one embodiment, the query statement for obtaining structured exported data includes:

[0092] Based on the table name, field names, and data structure, obtain the query statement for structured data export.

[0093] In one embodiment, the parsing of the unstructured exported data based on the data structure of the unstructured exported data includes:

[0094] The location of the unstructured exported data in the data table is obtained based on the data address, and the corresponding length of unstructured exported data is read from the starting data.

[0095] In one embodiment, the export tool based on the database to be transferred generates structured export data and unstructured export data, including:

[0096] Select the appropriate export tool for structured and unstructured data based on the type of database to be transferred.

[0097] In one embodiment, Figure 5 This is a structural block diagram of a data migration device in one embodiment, wherein the import module includes:

[0098] The processing module 502 submits a batch import data request to the target database for the query statement, and performs a merged processing on the batch import data request to assemble it into a single query statement.

[0099] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0100] The modules described above in the data migration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0101] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data to be transferred. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data migration method.

[0102] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. The computer program, when executed by the processor, implements a data migration method. The display screen can be an LCD screen or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0103] Those skilled in the art will understand that Figure 6 , Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0104] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0105] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0107] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A data migration method, characterized by, The method comprises: In response to a data migration instruction, obtaining to-be-migrated data in a to-be-migrated database, generating structured export data and unstructured export data based on an export tool of the to-be-migrated database; According to a data table table name and a field name in configuration information, columnizing each row of structured export data in the data table to obtain a data structure of each structured export data, the data structure comprising a position, an order, and a value of the structured export data, and obtaining a query statement of the structured export data based on the data table table name, the field name, and the data structure, the configuration information comprising the data table table name and the field name; According to a data address, obtaining a position of unstructured export data in the data table, reading unstructured export data of a corresponding data length from a starting data, and obtaining a query statement of the unstructured export data, the data structure comprising the starting data, the data length, and the data address; Splicing the query statement of the structured export data and the query statement of the unstructured export data, and importing the to-be-migrated data into a target database according to the spliced query statement.

2. The method of claim 1, wherein, Based on the export tool of the to-be-migrated database, generating the structured export data and the unstructured export data comprises: Selecting an export tool corresponding to the structured export data and the unstructured export data respectively based on a type of the to-be-migrated database.

3. The method of claim 1, wherein, The importing of the to-be-migrated data into the target database according to the spliced query statement comprises: Making a data request for batch importing of the target database to the query statement, and performing assembly and merging processing on the batch data request to form one query statement.

4. A data migration apparatus, characterized by comprising: The apparatus comprises: An export module configured to, in response to a data migration instruction, obtain to-be-migrated data in a to-be-migrated database, and generate structured export data and unstructured export data based on an export tool of the to-be-migrated database; A first analysis module configured to, according to a data table table name and a field name in configuration information, columnize each row of structured export data in the data table to obtain a data structure of each structured export data, the data structure comprising a position, an order, and a value of the structured export data, and obtain a query statement of the structured export data based on the data table table name, the field name, and the data structure, the configuration information comprising the data table table name and the field name; A second analysis module configured to, according to a data address, obtain a position of unstructured export data in the data table, read unstructured export data of a corresponding data length from a starting data, and obtain a query statement of the unstructured export data, the data structure comprising the starting data, the data length, and the data address; An import module configured to splice the query statement of the structured export data and the query statement of the unstructured export data, and import the to-be-migrated data into a target database according to the spliced query statement. 5.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-4 when the computer program is executed by the processor. The processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 3.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 3.

7. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 3.

Images