Data migration method, device, equipment and medium

Abstract

The present invention discloses a data migration method, device, equipment and medium. The method includes: creating a replica data table in a target database; detecting whether each data in each stock data table in the database to be migrated contains illegal characters, and outputting data information of abnormally coded data containing illegal characters into an abnormal data file; transcoding normal coded data and importing it into the corresponding replica data table, and importing abnormally coded data into the corresponding replica data table in hexadecimal format; detecting whether each data in each incremental data table contains illegal characters; transcoding data in each normal incremental data table and importing it into the corresponding replica data table, and importing data in each abnormal incremental data table into the corresponding replica data table in hexadecimal format. The embodiment of the present invention can process illegal characters in non-domestic databases, avoiding garbled characters and import failure error problems.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a data migration method, device, equipment and medium. Background Art

[0002] To safeguard the information security of financial institutions and ensure the security and controllability of their information systems, financial institutions will conduct a domestic migration of non-domestic databases installed within the financial institutions, migrating data from non-domestic databases to domestic databases. Non-domestic databases and domestic databases are heterogeneous databases. The character set used by non-domestic databases is different from the character set used by domestic databases. Data written to non-domestic databases is typically encoded according to the encoding rules of the character set used by the non-domestic database, and the characters in the data are all characters that comply with the encoding rules of the character set used by the non-domestic database. In special cases, data that is not encoded according to the encoding rules of the character set used by the non-domestic database will be written to the non-domestic database, and illegal characters will be contained in the data. Illegal characters are characters that do not comply with the encoding rules of the character set used by the non-domestic database.

[0003] In the related art, the commonly used data migration scheme is: uniformly transcode the data in the non-domestic database into data encoded according to the encoding rules of the specified character set and then import it into the domestic database, or directly uniformly import the data in the non-domestic database into the domestic database, and then the domestic database transcodes the imported data into data encoded according to the encoding rules of the character set used by the domestic database. In the process of uniformly transcoding the data and importing it into the domestic database, illegal characters will appear garbled during the transcoding process, resulting in data containing garbled characters being imported into the domestic database. After uniformly importing the data directly into the domestic database, the illegal characters will be verified by the domestic database to be non-existent, resulting in the domestic database failing to import and reporting an error. The data migration scheme in the related art cannot handle illegal characters in non-domestic databases, resulting in data containing garbled characters being imported into the domestic database or causing the domestic database to fail to import and reporting an error, and the correctness of the data migration cannot be guaranteed. Summary of the Invention

[0004] The present invention provides a data migration method, device, equipment and medium to solve the problem that the data migration scheme in the related art cannot process illegal characters in non-domestic databases, resulting in data containing garbled characters being imported into domestic databases or causing the domestic database to fail to import and report an error, and the correctness of the data migration cannot be guaranteed.

[0005] According to one aspect of the present invention, a data migration method is provided, comprising:

[0006] Determine each existing data table and each incremental data table in the database to be migrated, and create a copy data table of each existing data table and each incremental data table in a target database corresponding to the database to be migrated;

[0007] Check whether each data in each existing data table in the database to be migrated contains illegal characters, and output the data information of each abnormally encoded data containing illegal characters to the abnormal data file;

[0008] According to the abnormal data file, the normal coded data that does not contain illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormal coded data that contains illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal format;

[0009] Detecting whether each data in each incremental data table in the database to be migrated contains illegal characters, configuring the incremental data table containing abnormally encoded data containing illegal characters as an abnormal incremental data table, and configuring the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table;

[0010] The data in each normal incremental data table is transcoded and imported into the corresponding replica data table, and the data in each abnormal incremental data table is imported into the corresponding replica data table in hexadecimal format.

[0011] According to another aspect of the present invention, there is provided a data migration device, comprising:

[0012] A data table determination module is used to determine each existing data table and each incremental data table in the database to be migrated, and create a copy data table of each existing data table and each incremental data table in the target database corresponding to the database to be migrated;

[0013] The stock data detection module is used to detect whether the data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal encoded data containing illegal characters detected to the abnormal data file;

[0014] The stock data migration module is used to, based on the abnormal data file, transcode the normal coded data that does not contain illegal characters in each detected stock data table and import it into the corresponding copy data table, and import the abnormal coded data that contains illegal characters in each detected stock data table in hexadecimal format into the corresponding copy data table;

[0015] An incremental data detection module is used to detect whether the data in each incremental data table in the database to be migrated contains illegal characters, and configure the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table;

[0016] The incremental data migration module is used to transcode the data in each normal incremental data table and import it into the corresponding replica data table, and import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal format.

[0017] According to another aspect of the present invention, an electronic device is provided, comprising:

[0018] at least one processor;

[0019] and a memory communicatively coupled to the at least one processor;

[0020] The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data migration method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data migration method described in any embodiment of the present invention when executed.

[0022] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the data migration method according to any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention is to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in the target database corresponding to the database to be migrated; then detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to an abnormal data file; according to the abnormal data file, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, and the abnormal coded data containing illegal characters in each stock data table detected is imported into the corresponding copy data table in hexadecimal; then detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormal coded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormal coded data containing illegal characters as a normal incremental data table; finally, The data in the data table is transcoded and imported into the corresponding copy data table, and the data in each abnormal incremental data table is imported into the corresponding copy data table in hexadecimal. This solves the problem that the data migration solution in the relevant technology cannot process illegal characters in non-domestic databases, resulting in data containing garbled characters being imported into domestic databases or causing domestic databases to fail to import and report errors, and the correctness of data migration cannot be guaranteed. The system can automatically detect illegal characters in the existing data and incremental data in non-domestic databases. Based on the illegal character detection results of the existing data, the data containing illegal characters in the existing data are imported into the domestic database in hexadecimal. Based on the illegal character detection results of the incremental data, the data in each incremental data table with abnormally encoded data containing illegal characters are imported into the domestic database in hexadecimal, avoiding garbled characters and import failure errors. It can process illegal characters in non-domestic databases, avoid importing data containing garbled characters into domestic databases or causing domestic databases to fail to import and report errors, and ensure the correctness of data migration.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 This is a flowchart of a data migration method provided in Example 1 of the present invention.

[0027] Figure 2 This is a flow chart of a data migration method provided in Example 2 of the present invention.

[0028] Figure 3 This is a structural diagram of a data migration device provided in Example 3 of the present invention.

[0029] Figure 4 A schematic diagram of the structure of an electronic device for implementing the data migration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "target", "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprise", "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Example 1

[0033] Figure 1 This is a flow chart of a data migration method provided in the first embodiment of the present invention. This embodiment is applicable to the case of migrating data from a non-domestic database to a domestic database. The method can be executed by a data migration device, which can be implemented in the form of hardware and / or software, and the data migration device can be configured in an electronic device. The electronic device can be an electronic device set up in a financial institution for migrating data from a non-domestic database to a domestic database. Figure 1 As shown, the method includes:

[0034] Step 101: Determine each existing data table and each incremental data table in the database to be migrated, and create a copy data table of each existing data table and each incremental data table in a target database corresponding to the database to be migrated.

[0035] Optionally, the database to be migrated can be a non-Chinese database set up at a financial institution. To ensure the financial institution's information security, data must be migrated from the database to be migrated to a Chinese database. The target database corresponding to the database to be migrated can be a Chinese database set up at the financial institution to replace the database to be migrated. The database to be migrated and the target database are heterogeneous databases. The character set used by the database to be migrated and the character set used by the target database are different.

[0036] Under normal circumstances, data written to the database to be migrated is encoded according to the encoding rules of the character set used by the database to be migrated. All characters in data encoded according to the encoding rules of the character set used by the database to be migrated comply with the encoding rules of the character set used by the database to be migrated. However, in special cases, data not encoded according to the encoding rules of the character set used by the database to be migrated may be written to the database to be migrated. Data not encoded according to the encoding rules of the character set used by the database to be migrated may contain illegal characters. Illegal characters are characters that do not comply with the encoding rules of the character set used by the database to be migrated. For example, the character set used by the target database can be the 8-bit Unicode Transformation Format (UTF-8) character set, or the information technology Chinese coded character set GB18030_2022. The database to be migrated can be an Oracle database (Oracle), and the character set used by the database to be migrated can be the multilingual character set AL32UTF8, or the simplified Chinese character set ZHS16GBK.

[0037] The UTF-8 character set can be used to represent any character in the international standard character set Unicode. The UTF-8 character set encoding rules are for variable-length characters, with encoding values ​​ranging from 1 to 4 bytes. The international standard character set Unicode is a character set used to unify the characters of various languages ​​​​in the world. The encoding range of the international standard character set Unicode is 0x0000-0x10FFFF. The international standard character set Unicode is compatible with the American Standard Code for Information Interchange (ASCII) character set and can accommodate over one million characters. The international standard character set Unicode has 17 planes. Plane 0 (U+0000-U+FFFF) is the Basic Multilingual Plane, and characters used daily throughout the world are in Plane 0. Characters in other planes (U+10000-U+10FFFF) are supplementary characters.

[0038] Exemplarily, the first character set and the second character set are two different character sets. When the encoding value of the first character set contains all the encoding values ​​of the second character set, and the same encoding values ​​of the first character set and the second character set represent the same character, it can be determined that the first character set is compatible with the second character set. The UTF-8 character set is compatible with the ASCII character set. The Chinese encoding character set GB18030_2022 of information technology is compatible with the Chinese Internal Code Specification (GBK) character set. The UTF-8 character set is compatible with the multi-language character set AL32UTF8. The Chinese encoding character set GB18030_2022 of information technology is compatible with the simplified Chinese character set ZHS16GBK. The multi-language character set AL32UTF8 can be equivalent to the UTF-8 character set. The simplified Chinese character set ZHS16GBK adds specific symbols relative to the GBK character set.

[0039] Optionally, the data in the database to be migrated includes existing data and incremental data. Existing data may refer to data written to the database to be migrated before the data migration process. Incremental data may refer to data newly written during the data migration process. Each existing data table in the database to be migrated may refer to multiple data tables in the database to be migrated for storing existing data of the database to be migrated. The existing data of the database to be migrated may refer to data written to the database to be migrated before the data migration process. Each incremental data table in the database to be migrated may refer to multiple data tables in the database to be migrated for storing incremental data of the database to be migrated.

[0040] Optionally, the name of a data table can be a string that is set according to the function or location of the data table and is used to uniquely identify the data table. The names of different data tables are different. For each existing data table, a copy data table of the existing data table is a data table created in the target database with the same name and format as the existing data table. For each incremental data table, a copy data table of the incremental data table is a data table created in the target database with the same name and format as the incremental data table. Based on the names of the existing data tables and the names of the incremental data tables in the database to be migrated, the existing data tables and the incremental data tables in the database to be migrated can be determined, and based on the names and formats of the existing data tables and the incremental data tables in the database to be migrated, a copy data table of each existing data table and each incremental data table can be created in the target database corresponding to the database to be migrated. The names of the existing data tables and the names of the incremental data tables in the database to be migrated can be sent by a technician.

[0041] Step 102: Detect whether each data item in each stock data table in the database to be migrated contains illegal characters, and output data information of each abnormal coded data item containing illegal characters detected to an abnormal data file.

[0042] Optionally, detecting whether each data item in each stock data table in the database to be migrated contains illegal characters includes: using an illegal character detection tool to read each data item in each stock data table in the database to be migrated in hexadecimal format and detecting whether each data item contains illegal characters.

[0043] Therefore, based on the illegal character detection tool, it is possible to automatically read and detect in hexadecimal whether the data in each existing data table in the database to be migrated contains illegal characters, and obtain the illegal character detection results of each data, which is convenient for subsequent processing of each data containing illegal characters based on the illegal character detection results of each data.

[0044] Optionally, illegal characters refer to characters that do not conform to the encoding rules of the character set used by the database to be migrated. The illegal character detection tool can be a software module for reading data in hexadecimal and detecting whether the data contains illegal characters. Reading data in hexadecimal and detecting whether the data contains illegal characters includes: reading data in hexadecimal, thereby converting the data into a hexadecimal string; detecting whether the hexadecimal string contains illegal characters; if the hexadecimal string contains illegal characters, it can be determined that the data contains illegal characters and the data is abnormally encoded data containing illegal characters; if the hexadecimal string does not contain illegal characters, it can be determined that the data does not contain illegal characters and the data is normally encoded data that does not contain illegal characters. Normally encoded data can refer to data that does not contain illegal characters and is encoded according to the encoding rules of the character set used by the database to be migrated. Abnormally encoded data can refer to data that contains illegal characters and is not encoded according to the encoding rules of the character set used by the database to be migrated.

[0045] Optionally, the various stock data tables in the database to be migrated are arranged in a preset order. The preset order can be the arrangement order of the various stock data tables specified by the technician. Starting from the stock data table ranked first, the illegal character detection tool can be used to read the data in each stock data table in hexadecimal and detect whether the data contains illegal characters, until the processing of the stock data table ranked last is completed. For each stock data table, the illegal character detection tool can be used to perform the following operations: read the data in the stock data table in hexadecimal, thereby converting the data in the stock data table into a hexadecimal string; detect whether each hexadecimal string contains illegal characters; for each hexadecimal string detected that contains illegal characters, determine that the data to which the hexadecimal string belongs contains illegal characters, and the data is abnormally encoded data containing illegal characters; for each hexadecimal string detected that does not contain illegal characters, determine that the data to which the hexadecimal string belongs does not contain illegal characters, and the data is normally encoded data that does not contain illegal characters. The special character processing function utl_raw.cast_to_raw can be used to read the data in the data table in hexadecimal.

[0046] Optionally, the data information of each detected abnormal coded data containing illegal characters is output to the abnormal data file, including: for each detected abnormal coded data containing illegal characters in the stock data table, the stock data table name, identifier and field name of the abnormal coded data are determined as the data information of the abnormal coded data, and the data information of the abnormal coded data is output to the abnormal data file.

[0047] In this way, the stock data table names, identifiers and field names of the data containing illegal characters in the stock data table can be automatically output to the abnormal data file, so as to facilitate the subsequent import of the data containing illegal characters into the target database based on the stock data table names, identifiers and field names of the data containing illegal characters.

[0048] Optionally, the data information of the abnormal coding data may include the stock data table name (TABLE), identifier (ROWID) and field name (COLUMN) of the abnormal coding data. The stock data table name of the abnormal coding data may refer to the name of the stock data table where the abnormal coding data is located. The identifier of the abnormal coding data may be information for identifying the position of the abnormal coding data in the stock data table. The field name of the abnormal coding data may refer to the name of the field where the abnormal coding data is located. The stock data table name, identifier and field name of the abnormal coding data can be extracted from the stock data table where the abnormal coding data is located. The data information of the abnormal coding data may also include the field type (TYPE), field length (SIZE), number of characters (CHAR_CNT), number of original bytes (CHAR_CNT), number of converted bytes (TGT_BYTES), number of illegal characters (LOSSY_CNT), abnormal data type (EXCEPTION), and data (DATA) of the abnormal coding data.

[0049] Optionally, the field type of the abnormally coded data may be information used to identify the type of the field containing the abnormally coded data. The field length of the abnormally coded data may be the upper limit of the length of data that can be stored in the field containing the abnormally coded data. The number of characters in the abnormally coded data may be the number of characters contained in the abnormally coded data. The number of original bytes of the abnormally coded data may be the number of bytes occupied by the abnormally coded data in the database to be migrated. The number of converted bytes of the abnormally coded data may be the number of bytes occupied by the abnormally coded data converted into a hexadecimal string in the database to be migrated. The number of illegal characters in the abnormally coded data may be the number of illegal characters contained in the abnormally coded data. The abnormal data type of the abnormally coded data may be information used to identify the type of abnormal problem that occurs after the abnormally coded data is converted into a hexadecimal string. If the abnormal data type of the abnormally coded data is "TRUNCATE", this indicates that the abnormal problem that occurs after the abnormally coded data is converted into a hexadecimal string is that the converted hexadecimal string exceeds the upper limit of the length. If the abnormal data type of the abnormally coded data is "CONVERT_LOSSY", this indicates that the abnormal problem that occurs after the abnormally coded data is converted into a hexadecimal string is that garbled characters appear in the converted hexadecimal string. The abnormally coded data can be a hexadecimal string obtained by reading the first 30 characters of the abnormally coded data in hexadecimal format. The abnormally coded data, the existing data table containing the abnormally coded data, and the hexadecimal string obtained after the abnormally coded data is converted can be analyzed and statistically analyzed to determine the field type, field length, number of characters, number of original bytes, number of converted bytes, number of illegal characters, abnormal data type, and data of the abnormally coded data.

[0050] Optionally, for each detected abnormally coded data containing illegal characters in the stock data table, the stock data table name, identifier, and field name of the abnormally coded data can be extracted from the stock data table where the abnormally coded data is located, thereby determining the stock data table name, identifier, and field name of the abnormally coded data, and then determining the stock data table name, identifier, and field name of the abnormally coded data as data information of the abnormally coded data, and outputting the data information of the abnormally coded data to an abnormal data file. The abnormal data file can be a pre-set file for storing data information of the abnormally coded data.

[0051] Step 103: According to the abnormal data file, the normally encoded data that does not contain illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormally encoded data that contains illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal.

[0052] Optionally, after outputting the data information of all detected abnormal coded data to the abnormal data file, according to the abnormal data file, the normal coded data that does not contain illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormal coded data that contains illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal.

[0053] Optionally, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, including: starting from the stock data table ranked first, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table in sequence, until the stock data table ranked last is processed. The normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, including: performing the following operations for each stock data table: detecting whether the data information of each data item in the stock data table is included in the abnormal data file, determining the data whose data information is not included in the abnormal data file as normal coded data that does not contain illegal characters; transcoding each normal coded data item that does not contain illegal characters, converting the character set of each normal coded data item that does not contain illegal characters from the character set used by the database to be migrated into an intermediate character set; and writing the transcoded normal coded data item that does not contain illegal characters into the copy data table of the stock data table where the normal coded data item that does not contain illegal characters is located. The data information not included in the abnormal data file is normal encoded data that does not contain illegal characters.

[0054] Optionally, each item of normally encoded data that does not contain illegal characters can be transcoded, and the normally encoded data can be converted from data encoded according to the encoding rules of the character set used by the database to be migrated to data encoded according to the encoding rules of the intermediate character set, thereby converting the character set of the normally encoded data from the character set used by the database to be migrated to the intermediate character set. The intermediate character set can be a character set that is pre-set and different from the character set used by the database to be migrated. Exemplarily, the intermediate character set is the UTF-8 character set. Each item of normally encoded data that does not contain illegal characters can be read through the Java Database Connection (JDBC) method of the non-domestic database, and the normally encoded data can be transcoded, and the normally encoded data can be converted from data encoded according to the encoding rules of the character set used by the database to be migrated to data encoded according to the encoding rules of the 16-bit Unicode Transformation Format (UTF-16) character set, thereby converting the character set of the normally encoded data from the character set used by the database to be migrated to the UTF-16 character set. Then, the JDBC method of the domestic database can be used to read the character set and convert it into normal encoded data of each item of the UTF-16 character set that does not contain illegal characters, transcode the normal encoded data, and convert the normal encoded data from data encoded according to the encoding rules of the UTF-16 character set into data encoded according to the encoding rules of the UTF-8 character set, thereby converting the character set of the normal encoded data from the UTF-16 character set to the UTF-8 character set.

[0055] Optionally, writing each item of transcoded normal coded data that does not contain illegal characters into a copy data table of the stock data table containing each item of normal coded data that does not contain illegal characters includes: performing the following operations for each item of normal coded data that does not contain illegal characters: determining, based on the name of the stock data table of the normal coded data, a copy data table of the stock data table containing the normal coded data in the target database; wherein a data table in the target database having the same name as the stock data table of the normal coded data is the copy data table of the stock data table containing the normal coded data in the target database; and writing, based on the identifier and field name of the normal coded data, the transcoded normal coded data into the copy data table of the stock data table containing the normal coded data, such that the position of the transcoded normal coded data in the copy data table of the stock data table containing the normal coded data is the same as the position of the normal coded data in the stock data table containing the normal coded data. The data table in the target database having the same name as the stock data table of the normal coded data can be determined as the copy data table of the stock data table containing the normal coded data in the target database. The identifier and field name of the normal coded data can be used to identify the position of the normal coded data in the stock data table containing the normal coded data. After determining the copy data table of the stock data table where the normally coded data is located in the target database, the transcoded normally coded data can be written to the copy data table of the stock data table where the normally coded data is located according to the identifier and field name of the normally coded data, so that the position of the transcoded normally coded data in the copy data table of the stock data table where the normally coded data is located is the same as the position of the normal coded data in the stock data table where the normally coded data is located.

[0056] Optionally, the abnormally coded data containing illegal characters detected in each stock data table is imported into a corresponding replica data table in hexadecimal format, including: reading data information of each abnormally coded data item in the abnormal data file; reading each abnormally coded data item in hexadecimal format based on the data information of each abnormally coded data item, and writing the read each abnormally coded data item into a replica data table of the stock data table where the abnormally coded data item is located. Typically, when the abnormally coded data item containing illegal characters is imported into a corresponding replica data table in the target database in hexadecimal format, garbled characters and import failure errors will not occur.

[0057] In this way, based on the data information of each abnormal coded data in the abnormal data file, the abnormal coded data containing illegal characters in each stock data table detected can be automatically imported into the target database in hexadecimal format, avoiding garbled characters and import failure errors.

[0058] Optionally, the data information of each abnormal coded data item in the abnormal data file can be read to obtain the stock data table name, identifier, field name, and other information of each abnormal coded data item. According to the data information of each abnormal coded data item, each abnormal coded data item is read in hexadecimal, and each abnormal coded data item read is written into a copy data table of the stock data table where each abnormal coded data item is located, including: performing the following operations on the acquired data information of each abnormal coded data item: determining the stock data table for storing the abnormal coded data item to which the data information belongs, and the position of the abnormal coded data item to which the data information belongs in the stock data table, based on the stock data table name, identifier, and field name in the data information; reading the abnormal coded data item to which the data information belongs in hexadecimal, thereby converting the abnormal coded data item to which the data information belongs into a hexadecimal string; and writing the abnormal coded data item to which the data information belongs, converted into a hexadecimal string, into a copy data table of the stock data table for storing the abnormal coded data item to which the data information belongs.

[0059] Optionally, the abnormal coded data belonging to the data information converted into a hexadecimal string is written into a copy data table of the stock data table used to store the abnormal coded data belonging to the data information, including: determining the copy data table of the stock data table in the target database for storing the abnormal coded data belonging to the data information based on the stock data table name in the data information; wherein, the data table in the target database with the same name as the stock data table name in the data information is the copy data table of the stock data table in the target database for storing the abnormal coded data belonging to the data information; based on the identifier and field name in the data information, the abnormal coded data belonging to the data information converted into a hexadecimal string is written into the copy data table of the stock data table used to store the abnormal coded data belonging to the data information, so that the position of the abnormal coded data belonging to the data information converted into a hexadecimal string in the copy data table of the stock data table used to store the abnormal coded data belonging to the data information is the same as the position of the abnormal coded data belonging to the data information in the stock data table.

[0060] Optionally, a data table having the same name as the stock data table in the data information in the target database can be determined as a copy data table of the stock data table in the target database for storing the abnormally coded data belonging to the data information. The identifier and field name in the data information can be used to identify the position of the abnormally coded data belonging to the data information in the stock data table. After determining the copy data table of the stock data table for storing the abnormally coded data belonging to the data information in the target database, the abnormally coded data belonging to the data information converted into a hexadecimal string can be written into the copy data table of the stock data table for storing the abnormally coded data belonging to the data information according to the identifier and field name in the data information through the character conversion function cast_to_varchar2, so that the position of the abnormally coded data belonging to the data information converted into a hexadecimal string in the copy data table of the stock data table for storing the abnormally coded data belonging to the data information is the same as the position of the abnormally coded data belonging to the data information in the stock data table.

[0061] Step 104: Detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table.

[0062] Optionally, an illegal character detection tool can be used to read each data item in each incremental data table in the database to be migrated in hexadecimal format and detect whether each data item contains illegal characters.

[0063] Optionally, each incremental data table in the database to be migrated is arranged in a target order. The target order can be the arrangement order of each incremental data table specified by the technician. Starting from the incremental data table ranked first, the illegal character detection tool can be used in turn to read the data in each incremental data table in hexadecimal and detect whether each data contains illegal characters, until the processing of the last incremental data table is completed. For each incremental data table, the illegal character detection tool can be used to perform the following operations: read the data in the incremental data table in hexadecimal, thereby converting the data in the incremental data table into a hexadecimal string; detect whether each hexadecimal string contains illegal characters; for each hexadecimal string detected that contains illegal characters, determine that the data to which the hexadecimal string belongs contains illegal characters, and the data is abnormally encoded data containing illegal characters; for each hexadecimal string detected that does not contain illegal characters, determine that the data to which the hexadecimal string belongs does not contain illegal characters, and the data is normally encoded data that does not contain illegal characters.

[0064] Optionally, the abnormal incremental data table may refer to an incremental data table in each incremental data table that stores abnormal coded data containing illegal characters. The abnormal incremental file may be a file for storing the name of the abnormal incremental data table. The normal incremental data table may refer to an incremental data table in each incremental data table that does not store abnormal coded data containing illegal characters. The normal incremental file may be a file for storing the name of the normal incremental data table. Configuring the incremental data table that contains abnormal coded data containing illegal characters as an abnormal incremental data table, and configuring the incremental data table that does not contain abnormal coded data containing illegal characters as a normal incremental data table, includes: for each incremental data table that contains abnormal coded data containing illegal characters, writing the name of the incremental data table into the abnormal incremental file; for each incremental data table that does not contain abnormal coded data containing illegal characters, writing the name of the incremental data table into the normal incremental file.

[0065] Step 105: Transcode the data in each normal incremental data table and import it into the corresponding replica data table; import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal format.

[0066] Optionally, the data in each normal incremental data table is transcoded and imported into the corresponding replica data table, and the data in each abnormal incremental data table is imported into the corresponding replica data table in hexadecimal, including: for each normal incremental data table, transcoding each data in the normal incremental data table, converting the character set of each data into an intermediate character set, arranging the transcoded data in chronological order of occurrence of each data, and obtaining data transactions of the normal incremental data table; for each abnormal incremental data table, reading each data in the abnormal incremental data table in hexadecimal, arranging the read data in chronological order of occurrence of each data, and obtaining data transactions of the abnormal incremental data table; writing the data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table to the target disk; generating a data queue for each normal incremental data table based on the data transactions of each normal incremental data table; generating a data queue for each abnormal incremental data table based on the data transactions of each abnormal incremental data table; writing the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table to the corresponding replica data table.

[0067] Therefore, based on the illegal character detection results of the incremental data table, the data in each incremental data table can be imported into the target database, the data in each incremental data table that does not contain abnormally encoded data containing illegal characters can be transcoded and imported into the corresponding copy data table, and the data in each incremental data table that contains abnormally encoded data containing illegal characters can be imported into the corresponding copy data table in hexadecimal, avoiding garbled code problems and import failure errors.

[0068] Optionally, for each normal incremental data table, each data item in the table can be read and transcoded. Data encoded according to the character set used by the database to be migrated can be converted to data encoded according to the intermediate character set. The character set of each data item can then be converted from the character set used by the database to be migrated to the intermediate character set. The transcoded data items can then be arranged in chronological order to form a data sequence. This data sequence represents the data transactions in the normal incremental data table. The data occurrence time can refer to the time the data was generated. The log of the database to be migrated stores the occurrence time of each data item. The occurrence time of each data item can be read from the log of the database to be migrated, and the transcoded data items can be arranged in chronological order to form a data sequence. The earliest occurrence time is arranged first, and the latest occurrence time is arranged last. The occurrence time of each data item can be read from the log of the database to be migrated using LogMiner, an analysis tool for the database to be migrated.

[0069] Optionally, for each abnormal incremental data table, each data item in the abnormal incremental data table can be read in hexadecimal format, thereby converting each data item in the abnormal incremental data table into a hexadecimal string. The converted hexadecimal strings are then arranged in chronological order of occurrence to form a data sequence. This data sequence is the data transaction in the abnormal incremental data table. The transaction with the earliest occurrence time is arranged first, and the transaction with the latest occurrence time is arranged last.

[0070] Optionally, the target disk may be a disk for storing data of the database to be migrated. The data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table are written to the target disk respectively.

[0071] Optionally, the data queue of a normal incremental data table may refer to a queue for storing data transactions of the normal incremental data table. Generating a data queue for each normal incremental data table based on the data transactions of each normal incremental data table includes: performing the following operations for each data transaction of the normal incremental data table: creating a queue for storing data transactions of the normal incremental data table; and writing the data transactions of the normal incremental data table to the created queue, thereby forming a data queue for the normal incremental data table.

[0072] Optionally, the data queue of the abnormal incremental data table may refer to a queue for storing data transactions of the abnormal incremental data table. Generating a data queue for each abnormal incremental data table based on the data transactions of each abnormal incremental data table includes: performing the following operations for each data transaction of the abnormal incremental data table: creating a queue for storing data transactions of the abnormal incremental data table; and writing the data transactions of the abnormal incremental data table to the created queue, thereby forming a data queue for the abnormal incremental data table.

[0073] Optionally, the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table are written into the corresponding replica data table, including: determining the data thread corresponding to each data queue; reading the data in each data queue in parallel through the data thread corresponding to each data queue, and writing the read data into the replica data table of the data table to which each data queue belongs.

[0074] In this way, data in each incremental data table can be imported into the target database in parallel based on multiple threads, thereby improving data migration efficiency.

[0075] Optionally, the data thread may be a thread for writing data in the queue to the data table. Determining the data thread corresponding to each data queue includes: allocating a data thread to each data queue of each normal incremental data table and each data queue of each abnormal incremental data table, thereby determining the data thread corresponding to each data queue.

[0076] Optionally, the replica data table of the data table to which each data queue belongs refers to a replica data table of the normal incremental data table or abnormal incremental data table to which each data queue belongs in the target database. Data in each data queue can be read in parallel by the data threads corresponding to each data queue, and the read data can be written to the replica data table of the data table to which each data queue belongs, thereby writing the transcoded data in each normal incremental data table to the corresponding replica data table in the target database, and writing the data in each abnormal incremental data table converted into hexadecimal strings to the corresponding replica data table in the target database.

[0077] Optionally, the data format of the transcoded normal coded data is data encoded according to the encoding rules of the intermediate character set. After the normal coded data that does not contain illegal characters is transcoded and imported into the corresponding copy data table, the target database can determine that the transcoded normal coded data is normal coded data that does not contain illegal characters encoded according to the encoding rules of the intermediate character set based on the data format of the transcoded normal coded data. If the character library of the target database is not the intermediate character set, the target database can transcode each item of normal coded data that does not contain illegal characters encoded according to the encoding rules of the intermediate character set, and convert the normal coded data that does not contain illegal characters encoded according to the encoding rules of the intermediate character set from data encoded according to the encoding rules of the intermediate character set to data encoded according to the encoding rules of the character set used by the target database, thereby converting the character set of the normal coded data that does not contain illegal characters from the intermediate character set to the character set used by the target database.

[0078] Optionally, the data format of the abnormally encoded data converted to a hexadecimal string is a hexadecimal string. After the abnormally encoded data containing illegal characters is imported into the corresponding replica data table in hexadecimal format, the target database can determine, based on the data format, that the data is abnormally encoded data containing illegal characters converted to a hexadecimal string. The target database can transcode each item of abnormally encoded data containing illegal characters converted to a hexadecimal string, converting the abnormally encoded data containing illegal characters converted to a hexadecimal string into data encoded according to the encoding rules of the character set used by the target database.

[0079] The technical solution of the embodiment of the present invention is to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in the target database corresponding to the database to be migrated; then detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to an abnormal data file; according to the abnormal data file, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, and the abnormal coded data containing illegal characters in each stock data table detected is imported into the corresponding copy data table in hexadecimal; then detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormal coded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormal coded data containing illegal characters as a normal incremental data table; finally, The data in the data table is transcoded and imported into the corresponding copy data table, and the data in each abnormal incremental data table is imported into the corresponding copy data table in hexadecimal. This solves the problem that the data migration solution in the relevant technology cannot process illegal characters in non-domestic databases, resulting in data containing garbled characters being imported into domestic databases or causing domestic databases to fail to import and report errors, and the correctness of data migration cannot be guaranteed. The system can automatically detect illegal characters in the existing data and incremental data in non-domestic databases. Based on the illegal character detection results of the existing data, the data containing illegal characters in the existing data are imported into the domestic database in hexadecimal. Based on the illegal character detection results of the incremental data, the data in each incremental data table with abnormally encoded data containing illegal characters are imported into the domestic database in hexadecimal, avoiding garbled characters and import failure errors. It can process illegal characters in non-domestic databases, avoid importing data containing garbled characters into domestic databases or causing domestic databases to fail to import and report errors, and ensure the correctness of data migration.

[0080] Example 2

[0081] Figure 2 This is a flow chart of a data migration method provided by the second embodiment of the present invention. The embodiment of the present invention can be combined with each optional solution in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0082] Step 201: Determine each existing data table and each incremental data table in the database to be migrated, and create a copy data table of each existing data table and each incremental data table in the target database corresponding to the database to be migrated.

[0083] Step 202: Detect whether each data item in each stock data table in the database to be migrated contains illegal characters, and output data information of each abnormal coded data item containing illegal characters detected to an abnormal data file.

[0084] Step 203: According to the abnormal data file, the normally encoded data that does not contain illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormally encoded data that contains illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal.

[0085] Step 204: Detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table.

[0086] Step 205: For each normal incremental data table, transcode each data in the normal incremental data table, convert the character set of each data into an intermediate character set, arrange the transcoded data in chronological order of occurrence of each data, and obtain the data transactions of the normal incremental data table.

[0087] Step 206: For each abnormal incremental data table, read each data in the abnormal incremental data table in hexadecimal format, arrange the read data in chronological order according to the occurrence time of each data, and obtain the data transaction of the abnormal incremental data table.

[0088] Step 207: Write the data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table to the target disk.

[0089] Step 208: Generate a data queue for each normal incremental data table based on the data transactions of each normal incremental data table.

[0090] Step 209: Generate a data queue for each abnormal incremental data table based on the data transactions of each abnormal incremental data table.

[0091] Step 210: Write the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table into the corresponding replica data table.

[0092] The technical solution of the embodiment of the present invention can import the data in each incremental data table into the target database based on the illegal character detection results of the incremental data table, transcode the data in each incremental data table that does not contain abnormally encoded data containing illegal characters and import it into the corresponding copy data table, and import the data in each incremental data table that contains abnormally encoded data containing illegal characters into the corresponding copy data table in hexadecimal, avoiding garbled code problems and import failure error problems.

[0093] Example 3

[0094] Figure 3 This is a structural diagram of a data migration device provided by the third embodiment of the present invention. The device can be configured in an electronic device. Figure 3 As shown, the device includes: a data table determination module 301, an existing data detection module 302, an existing data migration module 303, an incremental data detection module 304 and an incremental data migration module 305.

[0095] Among them, the data table determination module 301 is used to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in the target database corresponding to the database to be migrated; the stock data detection module 302 is used to detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to the abnormal data file; the stock data migration module 303 is used to transcode the normal coded data that does not contain illegal characters in each stock data table detected according to the abnormal data file and import it into the corresponding copy data table , import the abnormal coded data containing illegal characters in each stock data table detected into the corresponding copy data table in hexadecimal; the incremental data detection module 304 is used to detect whether the data in each incremental data table in the database to be migrated contains illegal characters, and configure the incremental data table with abnormal coded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormal coded data containing illegal characters as a normal incremental data table; the incremental data migration module 305 is used to transcode the data in each normal incremental data table and import it into the corresponding copy data table, and import the data in each abnormal incremental data table into the corresponding copy data table in hexadecimal.

[0096] The technical solution of the embodiment of the present invention is to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in the target database corresponding to the database to be migrated; then detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to an abnormal data file; according to the abnormal data file, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, and the abnormal coded data containing illegal characters in each stock data table detected is imported into the corresponding copy data table in hexadecimal; then detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormal coded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormal coded data containing illegal characters as a normal incremental data table; finally, The data in the data table is transcoded and imported into the corresponding copy data table, and the data in each abnormal incremental data table is imported into the corresponding copy data table in hexadecimal. This solves the problem that the data migration solution in the relevant technology cannot process illegal characters in non-domestic databases, resulting in data containing garbled characters being imported into domestic databases or causing domestic databases to fail to import and report errors, and the correctness of data migration cannot be guaranteed. The system can automatically detect illegal characters in the existing data and incremental data in non-domestic databases. Based on the illegal character detection results of the existing data, the data containing illegal characters in the existing data are imported into the domestic database in hexadecimal. Based on the illegal character detection results of the incremental data, the data in each incremental data table with abnormally encoded data containing illegal characters are imported into the domestic database in hexadecimal, avoiding garbled characters and import failure errors. It can process illegal characters in non-domestic databases, avoid importing data containing garbled characters into domestic databases or causing domestic databases to fail to import and report errors, and ensure the correctness of data migration.

[0097] In an optional implementation of an embodiment of the present invention, optionally, when the stock data detection module 302 performs the operation of detecting whether each data in each stock data table in the database to be migrated contains illegal characters, it is specifically used to: use an illegal character detection tool to read each data in each stock data table in the database to be migrated in hexadecimal and detect whether each data contains illegal characters.

[0098] In an optional implementation of an embodiment of the present invention, optionally, when the stock data detection module 302 performs the operation of outputting the data information of each detected abnormal coded data containing illegal characters to the abnormal data file, it is specifically used to: for each detected abnormal coded data containing illegal characters in the stock data table, determine the stock data table name, identifier and field name of the abnormal coded data as the data information of the abnormal coded data, and output the data information of the abnormal coded data to the abnormal data file.

[0099] In an optional implementation of an embodiment of the present invention, optionally, when the stock data migration module 303 performs the operation of importing the detected abnormal coded data containing illegal characters in each stock data table into the corresponding copy data table in hexadecimal, it is specifically used to: read the data information of each abnormal coded data in the abnormal data file; read each abnormal coded data in hexadecimal according to the data information of each abnormal coded data, and write the read each abnormal coded data into the copy data table of the stock data table where the each abnormal coded data is located.

[0100] In an optional implementation of an embodiment of the present invention, optionally, the incremental data migration module 305 is specifically used to: for each normal incremental data table, transcode the data in the normal incremental data table, convert the character set of each data into an intermediate character set, and arrange the transcoded data in chronological order of occurrence of each data to obtain data transactions of the normal incremental data table; for each abnormal incremental data table, read the data in the abnormal incremental data table in hexadecimal, arrange the read data in chronological order of occurrence of each data to obtain data transactions of the abnormal incremental data table; write the data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table to the target disk; generate a data queue for each normal incremental data table based on the data transactions of each normal incremental data table; generate a data queue for each abnormal incremental data table based on the data transactions of each abnormal incremental data table; write the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table to the corresponding copy data table.

[0101] In an optional implementation of an embodiment of the present invention, optionally, when the incremental data migration module 305 performs the operation of writing the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table into the corresponding replica data table, it is specifically used to: determine the data thread corresponding to each data queue; read the data in each data queue in parallel through the data thread corresponding to each data queue, and write the read data into the replica data table of the data table to which each data queue belongs.

[0102] The data migration device provided in the embodiment of the present invention can execute the data migration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0103] Example 4

[0104] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement the data migration method of an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0105] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0107] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data migration method.

[0108] In some embodiments, the data migration method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on a heterogeneous hardware accelerator via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the data migration method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform the data migration method by any other appropriate means (e.g., by means of firmware).

[0109] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0111] In the context of the present invention, computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as data electronics), or a computing system that includes middleware components (e.g., application electronics), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] A computing system may include a client and an electronic device. The client and electronic device are generally remote from each other and typically interact via a communication network. The client-electronic device relationship is established by computer programs running on the respective computers and establishing a client-electronic device relationship. The electronic device may be a cloud electronic device, also known as a cloud computing electronic device or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited business scalability of traditional physical hosts and VPS services.

[0115] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0116] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data migration method, characterized in that: include: Determine each existing data table and each incremental data table in the database to be migrated, and create a copy data table of each existing data table and each incremental data table in a target database corresponding to the database to be migrated; Check whether each data in each existing data table in the database to be migrated contains illegal characters, and output the data information of each abnormally encoded data containing illegal characters to the abnormal data file; According to the abnormal data file, the normal coded data that does not contain illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormal coded data that contains illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal format; Detecting whether each data in each incremental data table in the database to be migrated contains illegal characters, configuring the incremental data table containing abnormally encoded data containing illegal characters as an abnormal incremental data table, and configuring the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table; The data in each normal incremental data table is transcoded and imported into the corresponding replica data table, and the data in each abnormal incremental data table is imported into the corresponding replica data table in hexadecimal format.

2. The data migration method according to claim 1, characterized in that: Check whether the data in each existing data table in the database to be migrated contains illegal characters, including: Use the illegal character detection tool to read the data in each existing data table in the database to be migrated in hexadecimal format and check whether the data contains illegal characters.

3. The data migration method according to claim 1, wherein: Output the data information of each abnormal encoded data containing illegal characters detected to the abnormal data file, including: For each abnormal coded data containing illegal characters detected in the stock data table, the stock data table name, identifier and field name of the abnormal coded data are determined as data information of the abnormal coded data, and the data information of the abnormal coded data is output to the abnormal data file.

4. The data migration method according to claim 1, wherein: Import the abnormally encoded data containing illegal characters in each inventory data table detected into the corresponding copy data table in hexadecimal format, including: Read the data information of each abnormal coded data in the abnormal data file; According to the data information of each abnormal coding data, each abnormal coding data is read in hexadecimal, and the read each abnormal coding data is written into the copy data table of the stock data table where the each abnormal coding data is located.

5. The data migration method according to claim 1, wherein: Transcode the data in each normal incremental data table and import it into the corresponding replica data table. Import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal format, including: For each normal incremental data table, transcode each data item in the normal incremental data table, convert the character set of each data item into an intermediate character set, and arrange the transcoded data items in chronological order to obtain the data transactions of the normal incremental data table; For each abnormal incremental data table, read each data in the abnormal incremental data table in hexadecimal format, arrange the read data in chronological order according to the occurrence time of each data, and obtain the data transaction of the abnormal incremental data table; Write the data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table to the target disk; Generate data queues for each normal incremental data table based on the data transactions of each normal incremental data table; Generate data queues for each abnormal incremental data table based on the data transactions of each abnormal incremental data table; The data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table are written into the corresponding replica data table.

6. The data migration method according to claim 5, characterized in that: Write the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table to the corresponding replica data table, including: Determine the data threads corresponding to each data queue; The data threads corresponding to the data queues are used to read the data in the data queues in parallel, and the read data are written to the replica data tables of the data tables to which the data queues belong.

7. A data migration device, characterized in that: include: A data table determination module is used to determine each existing data table and each incremental data table in the database to be migrated, and create a copy data table of each existing data table and each incremental data table in the target database corresponding to the database to be migrated; The stock data detection module is used to detect whether the data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal encoded data containing illegal characters detected to the abnormal data file; The stock data migration module is used to, based on the abnormal data file, transcode the normal coded data that does not contain illegal characters in each detected stock data table and import it into the corresponding copy data table, and import the abnormal coded data that contains illegal characters in each detected stock data table in hexadecimal format into the corresponding copy data table; An incremental data detection module is used to detect whether the data in each incremental data table in the database to be migrated contains illegal characters, and configure the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table; The incremental data migration module is used to transcode the data in each normal incremental data table and import it into the corresponding replica data table, and import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal format.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data migration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data migration method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the data migration method according to any one of claims 1 to 6.

Images