A data migration method, system, device and medium

Abstract

The application discloses a data migration method, system, device and medium, and the data migration method comprises the following steps: in response to received data migration trigger information, input parameters corresponding to the data migration trigger information are acquired; initial running files and initial database query scripts corresponding to the input parameters are acquired through a preset intermediate document page corresponding calling interface; and initial database query scripts and initial running files are respectively updated based on fields corresponding to the initial database query scripts, so that corresponding target database query scripts and target running files are generated. The intermediate document page is updated by using the target database query scripts and the target running files, a target document page is generated, data calling is performed through a target document page corresponding conversion interface, and migration data corresponding to the data migration trigger information is generated. By focusing on changes in the initial database query scripts, the initial database query scripts and the initial running files are updated, the learning cost is low, and the workload of manual checking is greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, and in particular to a data migration method, system, device and medium. Background Technology

[0002] As technology advances, older systems are gradually replaced by newer ones. Businesses often want to use the new system while retaining data from the old one, necessitating data migration to move all data from the old system to the new one.

[0003] Currently, the common data migration method involves configuring migration mappings for each field in a migration table to achieve the data migration effect. However, when migrating data from large and complex systems, which contain numerous migration tables, it is necessary to configure migration mappings for each field in each table individually. This requires executing the job one by one, and the configuration needs to be repeatedly modified during the data verification phase, resulting in a large workload for data migration.

[0004] Therefore, existing data migration methods suffer from drawbacks such as complex configuration steps and low data migration efficiency. Summary of the Invention

[0005] This invention provides a data migration method, system, device, and medium, which solves the technical problems of existing data migration methods, such as complicated configuration steps and low data migration efficiency.

[0006] The present invention provides a data migration method, comprising:

[0007] In response to the received data migration trigger information, obtain the input parameters corresponding to the data migration trigger information;

[0008] The initial running file and initial database query script corresponding to the input parameters are obtained through the calling interface corresponding to the preset intermediate document page;

[0009] Update the initial database query script and the initial running file according to the field mapping corresponding to the initial database query script, and generate the corresponding target database query script and target running file;

[0010] The intermediate document page is updated using the target database query script and the target runtime file to generate the target document page;

[0011] Data is retrieved through the conversion interface corresponding to the target document page to generate migration data corresponding to the data migration trigger information.

[0012] Optionally, before obtaining the initial conversion file and initial database query script corresponding to the input parameters through the calling interface corresponding to the preset intermediate document page, the method further includes:

[0013] Configure multiple data sources corresponding to the data migration trigger information using a preset file configuration method;

[0014] The preset initial document page is updated using all the data sources, and the preset intermediate document page is generated.

[0015] Optionally, the step of obtaining the initial running file and initial database query script corresponding to the input parameters through the calling interface corresponding to the preset intermediate document page includes:

[0016] The input parameters are obtained through the API call interface corresponding to the preset intermediate document page;

[0017] Based on the metadata migration table and the preset field mapping strategy, a migration table similarity set corresponding to the metadata migration table is generated;

[0018] The source table field and the target table field corresponding to the maximum field similarity in the migration table similarity set are selected and mapped to generate mapping data;

[0019] Using the mapping data and the migration configuration file corresponding to the migration table similarity set, the initial running file and initial database query script corresponding to the input parameters are constructed.

[0020] Optionally, the metadata migration table includes a source table and a target table; the preset field mapping strategy includes field annotation similarity mapping and field name similarity mapping; the step of generating a migration table similarity set corresponding to the metadata migration table based on the metadata migration table and the preset field mapping strategy includes:

[0021] Based on the field annotation similarity mapping, calculate the first field similarity between the source table field corresponding to the source table and the target table field corresponding to the target table;

[0022] Based on the field name similarity mapping, calculate the second field similarity between the corresponding fields in the source table and the target table respectively;

[0023] The similarity scores of all the first field and all the similarity scores of the second field are used as the migration table similarity set corresponding to the metadata migration table.

[0024] Optionally, the step of updating the initial database query script and the initial running file according to the field mapping corresponding to the initial database query script, and generating the corresponding target database query script and target running file, includes:

[0025] Determine whether the field mapping corresponding to the initial database query script meets the preset mapping standard;

[0026] If so, then the initial database query script and the initial running file will be used as the target database query script and the target running file, respectively;

[0027] If not, then based on the mapping error corresponding to the initial database query script, generate the corresponding target database query script and target runtime file.

[0028] Optionally, the step of generating a corresponding target database query script and a target runtime file based on the mapping error corresponding to the initial database query script includes:

[0029] Obtain the mapping error corresponding to the initial database query script and correct it to generate the corresponding target data query script;

[0030] The initial running file is updated using the target data query script to generate the corresponding target running file.

[0031] Optionally, the step of generating migration data corresponding to the data migration trigger information by calling data through the conversion interface corresponding to the target document page includes:

[0032] Obtain the file migration type data corresponding to the data migration trigger information;

[0033] The conversion interface corresponding to the target document page is used to migrate all target running files corresponding to the file migration type data, generating migration data corresponding to the data migration trigger information.

[0034] The present invention also provides a data migration system, comprising:

[0035] The input parameter acquisition module is used to respond to the received data migration trigger information and acquire the input parameters corresponding to the data migration trigger information.

[0036] The initial runtime file and initial database query script acquisition module is used to obtain the initial runtime file and initial database query script corresponding to the input parameters through the call interface corresponding to the preset intermediate document page;

[0037] The target database query script and target runtime file generation module is used to update the initial database query script and the initial runtime file respectively according to the field mapping corresponding to the initial database query script, and generate the corresponding target database query script and target runtime file;

[0038] The target document page generation module is used to update the intermediate document page using the target database query script and the target runtime file, and generate the target document page.

[0039] The migration data generation module is used to call data through the conversion interface corresponding to the target document page and generate migration data corresponding to the data migration trigger information.

[0040] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs steps to implement any of the data migration methods described above.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements any of the data migration methods described above.

[0042] As can be seen from the above technical solutions, the present invention has the following advantages:

[0043] This invention obtains the corresponding input parameters by responding to received data migration trigger information. It retrieves the initial runtime file and initial database query script corresponding to the input parameters through a pre-defined intermediate document page's calling interface. Based on the field mappings of the initial database query script, it updates both the initial database query script and the initial runtime file, generating the corresponding target database query script and target runtime file. The intermediate document page is then updated using the target database query script and target runtime file to generate the target document page. Data is then called through the conversion interface corresponding to the target document page to generate the migration data corresponding to the data migration trigger information. This invention solves the technical problems of existing data migration methods, such as complex configuration steps and low data migration efficiency. By focusing on changes in the initial database query script and updating the initial database query script and initial runtime file accordingly, the learning curve is low; users only need to focus on the initial database query script, significantly reducing manual verification workload. It also enables batch migration execution, improving migration efficiency. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the steps of a data migration method provided in Embodiment 1 of the present invention;

[0046] Figure 2 This is a flowchart of the steps of a data migration method provided in Embodiment 2 of the present invention;

[0047] Figure 3 This is a structural block diagram of a data migration system provided in Embodiment 3 of the present invention. Detailed Implementation

[0048] This invention provides a data migration method to address the technical problems of existing data migration methods, such as complex configuration steps and low data migration efficiency.

[0049] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0050] Please see Figure 1 , Figure 1 This is a flowchart of a data migration method provided in Embodiment 1 of the present invention.

[0051] The present invention provides a data migration method, comprising:

[0052] Step 101: Respond to the received data migration trigger information and obtain the input parameters corresponding to the data migration trigger information.

[0053] Input parameters include source data source name, target data source name, source table name, target table name, automatic mapping similarity threshold, file generation path, and other data.

[0054] In this embodiment of the invention, when data migration trigger information is received, data such as the source data source name, target data source name, source table name, target table name, automatic mapping similarity threshold, and file generation path corresponding to the data migration trigger information are obtained.

[0055] Step 102: Obtain the initial running file and initial database query script corresponding to the input parameters through the call interface corresponding to the preset intermediate document page.

[0056] The initial runtime files include the initial Kettle job file (kjb) and the initial Kettle transformation file (ktr). The initial database query script is an SQL file.

[0057] In this embodiment of the invention, multiple metadata migration tables corresponding to the input parameters are obtained through a pre-defined intermediate document page's API call. Based on the metadata migration tables and a pre-defined field mapping strategy, a migration table similarity set corresponding to the metadata migration tables is generated. The source table field and target table field corresponding to the maximum field similarity in the migration table similarity set are selected and mapped to generate mapping data. Using the mapping data and the migration configuration file corresponding to the migration table similarity set, the initial runtime file and initial database query script corresponding to the input parameters are constructed.

[0058] Step 103: Update the initial database query script and the initial running file according to the field mapping corresponding to the initial database query script, and generate the corresponding target database query script and target running file.

[0059] In this embodiment of the invention, it is determined whether the field mapping corresponding to the initial database query script meets the preset mapping standard. If yes, the initial database query script and the initial running file are used as the target database query script and the target running file, respectively. If not, the corresponding target database query script and target running file are generated based on the mapping error corresponding to the initial database query script. While keeping the program running, the SQL file is opened to check whether the field mapping is correct. Since the field mapping is matched based on similarity, there will be some errors that need to be corrected manually. After the correction is completed, the SQL is synchronously updated to the ktr file and kjb file.

[0060] Step 104: Update the intermediate document page using the target database query script and the target runtime file to generate the target document page.

[0061] In this embodiment of the invention, a target database query script and a target runtime file are used to update the intermediate document page to generate the corresponding target document page.

[0062] Step 105: Use the conversion interface corresponding to the target document page to call data and generate migration data corresponding to the data migration trigger information.

[0063] In this embodiment of the invention, file migration type data corresponding to the data migration trigger information is obtained, and all target running files corresponding to the file migration type data are migrated through the conversion interface corresponding to the target document page to generate migration data corresponding to the data migration trigger information.

[0064] In this embodiment of the invention, input parameters corresponding to the received data migration trigger information are obtained in response to the data migration trigger information. The initial running file and initial database query script corresponding to the input parameters are obtained through the call interface corresponding to the preset intermediate document page. Based on the field mapping corresponding to the initial database query script, the initial database query script and initial running file are updated respectively, generating the corresponding target database query script and target running file. The intermediate document page is updated using the target database query script and target running file to generate the target document page. Data is called through the conversion interface corresponding to the target document page to generate the migration data corresponding to the data migration trigger information. This solves the technical problems of existing data migration methods, such as complex configuration steps and low data migration efficiency. By focusing on changes in the initial database query script and updating the initial database query script and initial running file accordingly, the learning cost is low; users only need to focus on the initial database query script, significantly reducing the workload of manual verification. Batch execution of migrations is possible, improving migration efficiency.

[0065] Please see Figure 2 , Figure 2 This is a flowchart of a data migration method provided in Embodiment 1 of the present invention.

[0066] Another data migration method provided by the present invention includes:

[0067] Step 201: Respond to the received data migration trigger information and obtain the input parameters corresponding to the data migration trigger information.

[0068] In this embodiment of the invention, when data migration trigger information is received, the input parameters corresponding to the data migration trigger information are determined.

[0069] Step 202: Configure multiple data sources corresponding to the data migration trigger information using the preset file configuration method.

[0070] In this embodiment of the invention, the preset file configuration method refers to opening the application.yml file and configuring multiple data sources corresponding to the data migration trigger information. The program is then started after configuring the multiple data sources corresponding to the data migration trigger information in the application.yml file.

[0071] Step 203: Update the preset initial document page using all data sources to generate the preset intermediate document page.

[0072] In this embodiment of the invention, the preset initial document page refers to the initial configuration page corresponding to Kettle. After inputting key parameters such as the source data source name, target data source name, source table name, target table name, automatic mapping similarity threshold, and file generation path, the preset initial document page is updated, and a preset intermediate document page is generated.

[0073] Step 204: Obtain the initial running file and initial database query script corresponding to the input parameters through the call interface corresponding to the preset intermediate document page.

[0074] Further, step 204 may include the following sub-steps S11-S14:

[0075] S11. Obtain multiple metadata migration tables corresponding to the input parameters through the call interface corresponding to the preset intermediate document page.

[0076] S12. Based on the metadata migration table and the preset field mapping strategy, generate a migration table similarity set corresponding to the metadata migration table.

[0077] S13. Select the source table field and target table field corresponding to the maximum field similarity in the migration table similarity set and map them to generate mapping data.

[0078] S14. Using the migration configuration file corresponding to the mapping data and the migration table similarity set, construct the initial running file and initial database query script corresponding to the input parameters.

[0079] In this embodiment of the invention, the metadata migration table includes a source table and a target table. The metadata information of the two tables, namely the metadata migration table, is retrieved from the TABLES and COLUMNS tables in the information_schema database of the two data sources, using the source table name and target table name passed as input parameters. This metadata includes the table name, table comments, field names, and field comments. An interface is called to implement the logic for comparing field comment similarity and field similarity. All similarities are calculated using the StrUtil.similar(source table field, target table field) method from the Hutool utility library. The source table field with the highest similarity is then mapped to the target table field to generate mapping data. Using the mapping data and the migration configuration file corresponding to the similarity set in the migration table, the initial runtime file and initial database query script corresponding to the input parameters are constructed. Specifically, after the iteration is complete, an SQL file, a KTR file, and a KJB file are created respectively.

[0080] Furthermore, the metadata migration table includes a source table and a target table. The preset field mapping strategy includes field annotation similarity mapping and field name similarity mapping. Step S12 may include the following sub-steps S121-S123:

[0081] S121. According to the field annotation similarity mapping, calculate the first field similarity between the source table field corresponding to the source table and the target table field corresponding to the target table.

[0082] S122. Calculate the second field similarity between the source table fields and the target table fields according to the field name similarity mapping.

[0083] S123. Take all the similarities of the first field and all the similarities of the second field as the similarity set of the migration table corresponding to the metadata migration table.

[0084] In this embodiment of the invention, the preset field mapping strategy includes creating migration configurations based on field annotation similarity (i.e., field annotation similarity mapping) and creating migration configurations based on field name similarity (i.e., field name similarity mapping). The first field similarity between the source table fields and the target table fields under the field annotation similarity mapping strategy is calculated respectively. The second field similarity between the source table fields and the target table fields under the field name similarity mapping strategy is calculated respectively, and all first field similarities and all second field similarities are used as the migration table similarity set corresponding to the metadata migration table. By creating migration configurations through different strategies, and ensuring that the generated configurations match most fields, the workload of manual verification is significantly reduced.

[0085] Step 205: Update the initial database query script and the initial runtime file according to the field mapping corresponding to the initial database query script, and generate the corresponding target database query script and target runtime file.

[0086] Furthermore, step 205 may include the following sub-steps S21-S23:

[0087] S21. Determine whether the field mapping corresponding to the initial database query script meets the preset mapping standard.

[0088] S22. If so, then the initial database query script and the initial running file will be used as the target database query script and the target running file, respectively.

[0089] S23. If not, then generate the corresponding target database query script and target runtime file based on the mapping error corresponding to the initial database query script.

[0090] In this embodiment of the invention, the preset mapping standard refers to a preset field mapping relationship. While the program is running, the initial database query script (SQL file) is opened to check if the field mapping is correct. If correct, the initial database query script and the initial running file are used as the target database query script and the target running file, respectively. Since field mapping is matched based on similarity, if incorrect, the mapping error is corrected. After modification, changes to the SQL file in the migration configuration storage location are monitored to update the corresponding ktr and kjb files, thereby generating the corresponding target database query script and target running file.

[0091] Furthermore, step S23 may include the following sub-steps S231-S232:

[0092] S231. Obtain the mapping error corresponding to the initial database query script and correct it to generate the corresponding target data query script.

[0093] S232. Use the target data query script to update the initial running file and generate the corresponding target running file.

[0094] In this embodiment of the invention, the mapping error corresponding to the initial database query script is obtained and corrected to generate the corresponding target data query script. When the program detects a change in content, it parses the target data query script (i.e., the SQL file) to initialize relevant information, and then regenerates the ktr and kjb files, i.e., generates the corresponding target runtime file.

[0095] Step 206: Update the intermediate document page using the target database query script and the target runtime file to generate the target document page.

[0096] In this embodiment of the invention, the corresponding target document page is generated by updating the intermediate document page with the target database query script and the target running file.

[0097] Step 207: Use the conversion interface corresponding to the target document page to call data and generate migration data corresponding to the data migration trigger information.

[0098] Furthermore, step 207 may include the following sub-steps S31-S32:

[0099] S31. Obtain the file migration type data corresponding to the data migration trigger information.

[0100] S32. Through the conversion interface corresponding to the target document page, perform data migration on all target running files corresponding to the file migration type data, and generate migration data corresponding to the data migration trigger information.

[0101] In this embodiment of the invention, file migration type data refers to the file type for which data migration is performed, including kjb and ktr. When the file migration type data is ktr, all ktr files in the storage path will be read into memory through the conversion interface corresponding to the target document page, and the core function of Kettle will be called to perform data migration.

[0102] In this embodiment of the invention, the prior art directly uses Kettle's GUI for configuration, which requires creating drag-and-drop table input components and insert / update components, configuring data sources for the table input and insert / update components respectively, and manually organizing SQL statements and configuring mapping relationships, resulting in low execution efficiency. In contrast, this embodiment of the invention obtains the input parameters corresponding to the received data migration trigger information in response to the data migration trigger information. Multiple data sources corresponding to the data migration trigger information are configured using a preset file configuration method. A preset initial document page is updated using all data sources to generate a preset intermediate document page. The initial running file and initial database query script corresponding to the input parameters are obtained through the call interface corresponding to the preset intermediate document page. Based on the field mapping corresponding to the initial database query script, the initial database query script and initial running file are updated respectively to generate the corresponding target database query script and target running file. The intermediate document page is updated using the target database query script and target running file to generate the target document page. Data is called through the conversion interface corresponding to the target document page to generate the migration data corresponding to the data migration trigger information. By providing an interface for batch calling migration jobs, batch execution of migrations can be achieved, improving migration efficiency.

[0103] Please see Figure 3 , Figure 3 This is a structural block diagram of a data migration system provided in Embodiment 3 of the present invention.

[0104] The present invention provides a data migration system, comprising:

[0105] The input parameter acquisition module 301 is used to respond to the received data migration trigger information and acquire the input parameters corresponding to the data migration trigger information.

[0106] The initial runtime file and initial database query script acquisition module 302 is used to obtain the initial runtime file and initial database query script corresponding to the input parameters through the call interface corresponding to the preset intermediate document page.

[0107] The target database query script and target runtime file generation module 303 is used to update the initial database query script and initial runtime file respectively according to the field mapping corresponding to the initial database query script, and generate the corresponding target database query script and target runtime file.

[0108] The target document page generation module 304 is used to update the intermediate document page using the target database query script and the target runtime file, and generate the target document page.

[0109] The migration data generation module 305 is used to call data through the conversion interface corresponding to the target document page and generate migration data corresponding to the data migration trigger information.

[0110] Optionally, the system also includes:

[0111] The data source configuration module is used to configure multiple data sources corresponding to data migration trigger information using a preset file configuration method.

[0112] The preset intermediate document page generation module is used to update the preset initial document page using all data sources and generate the preset intermediate document page.

[0113] Optionally, the initial runtime file and initial database query script acquisition module 302 includes:

[0114] The metadata migration table acquisition module is used to obtain multiple metadata migration tables corresponding to the input parameters through the call interface corresponding to the preset intermediate document page.

[0115] The migration table similarity set generation module is used to generate a migration table similarity set corresponding to the metadata migration table based on the metadata migration table and the preset field mapping strategy.

[0116] The mapping data generation module is used to select the source table field and the target table field corresponding to the maximum field similarity in the migration table similarity set, and then map them to generate mapping data.

[0117] The initial runtime file and initial database query script acquisition submodule is used to construct the initial runtime file and initial database query script corresponding to the input parameters using the migration configuration file corresponding to the mapping data and the migration table similarity set.

[0118] Optionally, the metadata migration table includes a source table and a target table. Preset field mapping strategies include field annotation similarity mapping and field name similarity mapping. The migration table similarity set generation module can perform the following steps:

[0119] Based on the field annotation similarity mapping, calculate the first field similarity between the source table fields corresponding to the source table and the target table fields corresponding to the target table;

[0120] Based on the field name similarity mapping, calculate the second field similarity between the corresponding fields in the source table and the target table respectively;

[0121] The similarity scores of all first fields and all second fields are used as the similarity set of the migration table corresponding to the metadata migration table.

[0122] Optionally, the target database query script and target runtime file generation module 303 includes:

[0123] The field mapping judgment module is used to determine whether the field mapping corresponding to the initial database query script meets the preset mapping standard.

[0124] The first submodule for generating the target database query script and the target runtime file is used to, if so, use the initial database query script and the initial runtime file as the target database query script and the target runtime file, respectively.

[0125] The second submodule for generating the target database query script and the target runtime file is used to generate the corresponding target database query script and the target runtime file based on the mapping error corresponding to the initial database query script, if not otherwise.

[0126] Optionally, the target database query script and the target runtime file second generation submodule can perform the following steps:

[0127] Obtain and correct the mapping error corresponding to the initial database query script, and generate the corresponding target data query script;

[0128] The initial runtime file is updated using a target data query script to generate the corresponding target runtime file.

[0129] Optionally, the migration data generation module 305 includes:

[0130] The file migration type data acquisition module is used to acquire file migration type data corresponding to data migration trigger information.

[0131] The migration data generation submodule is used to migrate all target running files corresponding to the file migration type data through the conversion interface corresponding to the target document page, and generate migration data corresponding to the data migration trigger information.

[0132] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, it causes the processor to perform a data migration method as described in any of the above embodiments.

[0133] The memory can be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory has storage space for program code used to perform any of the method steps described above. For example, the storage space for program code may include individual program codes for implementing the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact discs (CDs), memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When run by a computing processing device, this code causes the computing processing device to perform the various steps in the data migration method described above.

[0134] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data migration method as described in any of the above embodiments.

[0135] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data migration method, characterized in that, include: In response to the received data migration trigger information, the system obtains the input parameters corresponding to the data migration trigger information; wherein, the input parameters include the source data source name, the target data source name, the source table name, the target table name, the automatic mapping similarity threshold, and the file generation path; The process involves obtaining the initial running file and initial database query script corresponding to the input parameters through a pre-defined intermediate document page's API call interface. This includes: obtaining multiple metadata migration tables corresponding to the input parameters through the pre-defined intermediate document page's API call interface; generating a migration table similarity set corresponding to the metadata migration tables based on the metadata migration tables and a pre-defined field mapping strategy; selecting the source table field and target table field corresponding to the maximum field similarity in the migration table similarity set for mapping, generating mapping data; and constructing the initial running file and initial database query script corresponding to the input parameters using the mapping data and the migration configuration file corresponding to the migration table similarity set. The initial running file includes the initial Kettle job file (kjb) and the transformation file (ktr). Update the initial database query script and the initial running file according to the field mapping corresponding to the initial database query script, and generate the corresponding target database query script and target running file; The intermediate document page is updated using the target database query script and the target runtime file to generate the target document page; Data is retrieved through the conversion interface corresponding to the target document page to generate migration data corresponding to the data migration trigger information.

2. The data migration method according to claim 1, characterized in that, Before obtaining the initial conversion file and initial database query script corresponding to the input parameters through the calling interface corresponding to the preset intermediate document page, the method further includes: Configure multiple data sources corresponding to the data migration trigger information using a preset file configuration method; The preset initial document page is updated using all the data sources, and the preset intermediate document page is generated.

3. The data migration method according to claim 1, characterized in that, The metadata migration table includes a source table and a target table; the preset field mapping strategy includes field annotation similarity mapping and field name similarity mapping; The step of generating a migration table similarity set corresponding to the metadata migration table based on the metadata migration table and a preset field mapping strategy includes: Based on the field annotation similarity mapping, calculate the first field similarity between the source table field corresponding to the source table and the target table field corresponding to the target table; Based on the field name similarity mapping, calculate the second field similarity between the corresponding fields in the source table and the target table respectively; The similarity scores of all the first field and all the similarity scores of the second field are used as the migration table similarity set corresponding to the metadata migration table.

4. The data migration method according to claim 1, characterized in that, The step of updating the initial database query script and the initial running file according to the field mapping corresponding to the initial database query script, and generating the corresponding target database query script and target running file, includes: Determine whether the field mapping corresponding to the initial database query script meets the preset mapping standard; If so, then the initial database query script and the initial running file will be used as the target database query script and the target running file, respectively; If not, then based on the mapping error corresponding to the initial database query script, generate the corresponding target database query script and target runtime file.

5. The data migration method according to claim 4, characterized in that, The step of generating a corresponding target database query script and a target executable file based on the mapping error corresponding to the initial database query script includes: Obtain the mapping error corresponding to the initial database query script and correct it to generate the corresponding target data query script; The initial running file is updated using the target data query script to generate the corresponding target running file.

6. The data migration method according to claim 1, characterized in that, The step of generating migration data corresponding to the data migration trigger information by calling data through the conversion interface corresponding to the target document page includes: Obtain the file migration type data corresponding to the data migration trigger information; The conversion interface corresponding to the target document page is used to migrate all target running files corresponding to the file migration type data, generating migration data corresponding to the data migration trigger information.

7. A data migration system, characterized in that, include: The input parameter acquisition module is used to respond to the received data migration trigger information and acquire the input parameters corresponding to the data migration trigger information; wherein, the input parameters include the source data source name, the target data source name, the source table name, the target table name, the automatic mapping similarity threshold, and the file generation path; The initial runtime file and initial database query script acquisition module is used to acquire the initial runtime file and initial database query script corresponding to the input parameters through a pre-defined intermediate document page's corresponding call interface. This includes: acquiring multiple metadata migration tables corresponding to the input parameters through the pre-defined intermediate document page's call interface; generating a migration table similarity set corresponding to the metadata migration tables based on the metadata migration tables and a pre-defined field mapping strategy; selecting the source table field and target table field corresponding to the maximum field similarity in the migration table similarity set for mapping, generating mapping data; and constructing the initial runtime file and initial database query script corresponding to the input parameters using the mapping data and the migration configuration file corresponding to the migration table similarity set. The initial runtime file includes the initial Kettle job file (kjb) and the transformation file (ktr). The target database query script and target runtime file generation module is used to update the initial database query script and the initial runtime file respectively according to the field mapping corresponding to the initial database query script, and generate the corresponding target database query script and target runtime file; The target document page generation module is used to update the intermediate document page using the target database query script and the target runtime file, and generate the target document page. The migration data generation module is used to call data through the conversion interface corresponding to the target document page and generate migration data corresponding to the data migration trigger information.

8. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the data migration method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the data migration method as described in any one of claims 1-6.

Images