Data migration method and device, equipment and medium

Abstract

The invention discloses a data migration method and device, equipment and a medium. The method comprises the following steps: creating a copy data table in a target database; detecting whether data in each stock data table in the to-be-migrated database contains illegal characters or not, and outputting data information of abnormal coded data containing the illegal characters to an abnormal data file; transcoding the normal coded data, importing the transcoded data into the corresponding copy data table, and importing the abnormal coded data into the corresponding copy data table in a hexadecimal mode; detecting whether each item of data in each incremental data table contains illegal characters or not; transcoding the data in each normal incremental data table, importing the transcoded data into the corresponding copy data table, and importing the data in each abnormal incremental data table into the corresponding copy data table in a hexadecimal mode. According to the embodiment of the invention, illegal characters in a non-domestic database can be processed, and the messy code problem and the error reporting problem caused by import failure are avoided.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a data migration method, device, equipment and medium. Background Art

[0002] In order to protect the information security of financial institutions and ensure that the information systems of financial institutions are safe and controllable, financial institutions will carry out localization migration for non-domestic databases set up in financial institutions, and migrate data from non-domestic databases to domestic databases. Non-domestic databases and domestic databases are heterogeneous databases. The character set used by non-domestic databases is different from the character set used by domestic databases. Data written to non-domestic databases are usually encoded according to the encoding rules of the character set used by non-domestic databases, and the characters in the data are all characters that comply with the encoding rules of the character set used by non-domestic databases. In special cases, data that is not encoded according to the encoding rules of the character set used by non-domestic databases will be written into non-domestic databases, and illegal characters will exist in the data. Illegal characters refer to characters that do not comply with the encoding rules of the character set used by non-domestic databases.

[0003] In the related art, the commonly used data migration scheme is: uniformly transcode the data in the non-domestic database into data encoded according to the encoding rules of the specified character set and then import it into the domestic database, or directly uniformly import the data in the non-domestic database into the domestic database, and then the domestic database transcodes the imported data into data encoded according to the encoding rules of the character set used by the domestic database. In the process of uniformly transcoding the data and importing it into the domestic database, illegal characters will appear garbled during the transcoding process, resulting in the import of garbled data into the domestic database. After uniformly importing the data directly into the domestic database, the illegal characters will be verified by the domestic database as not existing, resulting in the domestic database failing to import and reporting an error. The data migration scheme in the related art cannot handle illegal characters in non-domestic databases, resulting in the import of garbled data into the domestic database or causing the domestic database to fail to import and reporting an error, and the correctness of the data migration cannot be guaranteed. Summary of the invention

[0004] The present invention provides a data migration method, device, equipment and medium to solve the problem that the data migration scheme in the related art cannot process illegal characters in a non-domestic database, resulting in data containing garbled characters being imported into a domestic database or causing the domestic database to fail to import and report an error, and the correctness of the data migration cannot be guaranteed.

[0005] According to one aspect of the present invention, there is provided a data migration method, comprising:

[0006] Determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in a target database corresponding to the database to be migrated;

[0007] Detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to an abnormal data file;

[0008] According to the abnormal data file, the normal coded data without illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormal coded data containing illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal format;

[0009] Detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table;

[0010] The data in each normal incremental data table is transcoded and imported into the corresponding replica data table, and the data in each abnormal incremental data table is imported into the corresponding replica data table in hexadecimal format.

[0011] According to another aspect of the present invention, there is provided a data migration device, comprising:

[0012] A data table determination module, used to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in a target database corresponding to the database to be migrated;

[0013] The stock data detection module is used to detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to the abnormal data file;

[0014] The stock data migration module is used to, according to the abnormal data file, transcode the normal coded data that does not contain illegal characters in each detected stock data table and import it into the corresponding copy data table, and import the abnormal coded data that contains illegal characters in each detected stock data table in hexadecimal into the corresponding copy data table;

[0015] An incremental data detection module is used to detect whether each data in each incremental data table in the database to be migrated contains illegal characters, and configures the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configures the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table;

[0016] The incremental data migration module is used to transcode the data in each normal incremental data table and import it into the corresponding replica data table, and import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor;

[0019] and a memory communicatively coupled to the at least one processor;

[0020] The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data migration method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data migration method described in any embodiment of the present invention when executed.

[0022] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the data migration method described in any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention is to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in the target database corresponding to the database to be migrated; then detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each detected abnormal coded data containing illegal characters to an abnormal data file; according to the abnormal data file, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, and the abnormal coded data containing illegal characters in each stock data table detected is imported into the corresponding copy data table in hexadecimal; then detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormal coded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormal coded data containing illegal characters as a normal incremental data table; finally, each normal incremental The data in the data table is transcoded and imported into the corresponding copy data table, and the data in each abnormal incremental data table is imported into the corresponding copy data table in hexadecimal, which solves the problem that the data migration solution in the related technology cannot process illegal characters in the non-domestic database, resulting in the import of data containing garbled characters into the domestic database or causing the domestic database to fail to import and report errors, and the correctness of data migration cannot be guaranteed. Illegal character detection can be automatically performed on the existing data and incremental data in the non-domestic database. Based on the illegal character detection results of the existing data, the data containing illegal characters in the existing data is imported into the domestic database in hexadecimal. Based on the illegal character detection results of the incremental data, the data in each incremental data table with abnormally encoded data containing illegal characters is imported into the domestic database in hexadecimal, avoiding garbled characters and import failure errors. Illegal characters in non-domestic databases can be processed to avoid importing garbled characters into the domestic database or causing the domestic database to fail to import and report errors, thereby ensuring the correctness of data migration.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 The present invention provides a flowchart of a data migration method according to the first embodiment of the present invention.

[0027] Figure 2 A flowchart of a data migration method provided in Embodiment 2 of the present invention.

[0028] Figure 3 A schematic diagram of the structure of a data migration device provided in Embodiment 3 of the present invention.

[0029] Figure 4 A schematic diagram of the structure of an electronic device for implementing the data migration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "target", "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprise", "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flow chart of a data migration method provided in the first embodiment of the present invention. This embodiment is applicable to the case where data is migrated from a non-domestic database to a domestic database. The method can be executed by a data migration device, which can be implemented in the form of hardware and / or software, and the data migration device can be configured in an electronic device. The electronic device can be an electronic device set in a financial institution for migrating data from a non-domestic database to a domestic database. Figure 1 As shown, the method includes:

[0034] Step 101: determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in a target database corresponding to the database to be migrated.

[0035] Optionally, the database to be migrated may be a non-domestic database set up in a financial institution. In order to ensure the information security of the financial institution, data must be migrated from the database to be migrated to a domestic database. The target database corresponding to the database to be migrated may be a domestic database set up in the financial institution to replace the database to be migrated. The database to be migrated and the target database are heterogeneous databases. The character set used by the database to be migrated and the character set used by the target database are different.

[0036] Under normal circumstances, the data written into the database to be migrated will be encoded according to the encoding rules of the character set used by the database to be migrated. The characters in the data encoded according to the encoding rules of the character set used by the database to be migrated are all characters that comply with the encoding rules of the character set used by the database to be migrated. In special cases, data that is not encoded according to the encoding rules of the character set used by the database to be migrated will be written into the database to be migrated. Illegal characters will exist in the data that is not encoded according to the encoding rules of the character set used by the database to be migrated. Illegal characters refer to characters that do not comply with the encoding rules of the character set used by the database to be migrated. Exemplarily, the character set used by the target database can be an 8-bit Unicode Transformation Format (UTF-8) character set, or an information technology Chinese encoding character set GB18030_2022. The database to be migrated can be an Oracle database (Oracle database, referred to as Oracle), and the character set used by the database to be migrated can be a character set AL32UTF8 that supports multiple languages, or a simplified Chinese character set ZHS16GBK.

[0037] The UTF-8 character set can be used to represent any character of the international standard character set Unicode. The encoding rule of the UTF-8 character set is an encoding rule for variable-length characters, and the encoding value is 1-4 bytes. The international standard character set Unicode is a character set used to unify the characters of various languages ​​in the world. The encoding range of the international standard character set Unicode is 0x0000-0x10FFFF. The international standard character set Unicode is compatible with the American Standard Code for Information Interchange (ASCII) character set and can accommodate more than one million characters. The international standard character set Unicode has 17 planes. Plane 0 (U+0000-U+FFFF) is the basic multilingual plane, and the characters used daily around the world are in plane 0. The characters in other planes (U+10000-U+10FFFF) are supplementary characters.

[0038] Exemplarily, the first character set and the second character set are two different character sets. When the encoding value of the first character set contains all the encoding values ​​of the second character set, and the same encoding values ​​of the first character set and the second character set represent the same character, it can be determined that the first character set is compatible with the second character set. The UTF-8 character set is compatible with the ASCII character set. The Chinese encoding character set GB18030_2022 of information technology is compatible with the Chinese Internal Code Specification (GBK) character set. The UTF-8 character set is compatible with the multi-language character set AL32UTF8. The Chinese encoding character set GB18030_2022 of information technology is compatible with the simplified Chinese character set ZHS16GBK. The character set AL32UTF8 that supports multiple languages ​​can be equivalent to the UTF-8 character set. The simplified Chinese character set ZHS16GBK adds specific symbols relative to the GBK character set.

[0039] Optionally, the data in the database to be migrated includes stock data and incremental data. Stock data may refer to data written into the database to be migrated before the data migration process. Incremental data may refer to data newly written during the data migration process. Each stock data table in the database to be migrated may refer to multiple data tables in the database to be migrated for storing stock data of the database to be migrated. Stock data of the database to be migrated may refer to data written into the database to be migrated before the data migration process. Each incremental data table in the database to be migrated may refer to multiple data tables in the database to be migrated for storing incremental data of the database to be migrated.

[0040] Optionally, the name of the data table can be a string that is set according to the function or location of the data table and is used to uniquely identify the data table. The names of different data tables are different. For each existing data table, a copy data table of the existing data table is a data table created in the target database with the same name and format as the existing data table. For each incremental data table, a copy data table of the incremental data table is a data table created in the target database with the same name and format as the incremental data table. According to the names of the existing data tables and the names of the incremental data tables in the database to be migrated, the existing data tables and the incremental data tables in the database to be migrated can be determined, and according to the names and formats of the existing data tables and the incremental data tables in the database to be migrated, the copy data tables of the existing data tables and the incremental data tables can be created in the target database corresponding to the database to be migrated. The names of the existing data tables and the incremental data tables in the database to be migrated can be sent by a technician.

[0041] Step 102: Detect whether each data item in each stock data table in the database to be migrated contains illegal characters, and output data information of each detected abnormal coded data item containing illegal characters to an abnormal data file.

[0042] Optionally, detecting whether each data item in each stock data table in the database to be migrated contains illegal characters includes: using an illegal character detection tool to read each data item in each stock data table in the database to be migrated in hexadecimal and detecting whether each data item contains illegal characters.

[0043] Therefore, based on the illegal character detection tool, it is possible to automatically read and detect in hexadecimal whether the data in each stock data table in the database to be migrated contains illegal characters, and obtain the illegal character detection results for each data, so as to facilitate the subsequent processing of each data containing illegal characters based on the illegal character detection results for each data.

[0044] Optionally, illegal characters refer to characters that do not conform to the encoding rules of the character set used by the database to be migrated. The illegal character detection tool may be a software module for reading data in hexadecimal and detecting whether the data contains illegal characters. Reading data in hexadecimal and detecting whether the data contains illegal characters includes: reading data in hexadecimal, thereby converting the data into a hexadecimal string; detecting whether the hexadecimal string contains illegal characters; if the hexadecimal string contains illegal characters, it can be determined that the data contains illegal characters, and the data is abnormally encoded data containing illegal characters; if the hexadecimal string does not contain illegal characters, it can be determined that the data does not contain illegal characters, and the data is normally encoded data that does not contain illegal characters. Normally encoded data may refer to data that does not contain illegal characters and is encoded according to the encoding rules of the character set used by the database to be migrated. Abnormally encoded data may refer to data that contains illegal characters and is not encoded according to the encoding rules of the character set used by the database to be migrated.

[0045] Optionally, each stock data table in the database to be migrated is arranged in a preset order. The preset order may be the arrangement order of each stock data table specified by the technician. Starting from the first stock data table, the illegal character detection tool may be used to read each data in each stock data table in hexadecimal and detect whether each data contains illegal characters, until the processing of the last stock data table is completed. For each stock data table, the illegal character detection tool may be used to perform the following operations: read each data in the stock data table in hexadecimal, thereby converting each data in the stock data table into a hexadecimal string; detect whether each hexadecimal string contains illegal characters; for each hexadecimal string containing illegal characters detected, determine that the data to which the hexadecimal string belongs contains illegal characters, and the data is abnormally encoded data containing illegal characters; for each hexadecimal string not containing illegal characters detected, determine that the data to which the hexadecimal string belongs does not contain illegal characters, and the data is normal encoded data not containing illegal characters. The data in the data table may be read in hexadecimal through the special character processing function utl_raw.cast_to_raw.

[0046] Optionally, the data information of each detected abnormal coded data containing illegal characters is output to an abnormal data file, including: for each detected abnormal coded data containing illegal characters in the stock data table, the stock data table name, identifier and field name of the abnormal coded data are determined as the data information of the abnormal coded data, and the data information of the abnormal coded data is output to the abnormal data file.

[0047] In this way, the stock data table names, identifiers and field names of the data containing illegal characters in the stock data table can be automatically output to the abnormal data file, so as to facilitate the subsequent import of the data containing illegal characters into the target database based on the stock data table names, identifiers and field names of the data containing illegal characters.

[0048] Optionally, the data information of the abnormal coding data may include the stock data table name (TABLE), identifier (ROWID) and field name (COLUMN) of the abnormal coding data. The stock data table name of the abnormal coding data may refer to the name of the stock data table where the abnormal coding data is located. The identifier of the abnormal coding data may be information for identifying the position of the abnormal coding data in the stock data table. The field name of the abnormal coding data may refer to the name of the field where the abnormal coding data is located. The stock data table name, identifier and field name of the abnormal coding data can be extracted from the stock data table where the abnormal coding data is located. The data information of the abnormal coding data may also include the field type (TYPE) of the abnormal coding data, the field length (SIZE), the number of characters (CHAR_CNT), the number of original bytes (CHAR_CNT), the number of converted bytes (TGT_BYTES), the number of illegal characters (LOSSY_CNT), the abnormal data type (EXCEPTION), and the data (DATA).

[0049] Optionally, the field type of the abnormal coded data may be information for characterizing the type of the field where the abnormal coded data is located. The field length of the abnormal coded data may be the upper limit of the length of data that can be stored in the field where the abnormal coded data is located. The number of characters of the abnormal coded data may be the number of characters contained in the abnormal coded data. The number of original bytes of the abnormal coded data may be the number of bytes occupied by the abnormal coded data in the database to be migrated. The number of converted bytes of the abnormal coded data may be the number of bytes occupied by the abnormal coded data converted into a hexadecimal string in the database to be migrated. The number of illegal characters of the abnormal coded data may be the number of illegal characters contained in the abnormal coded data. The abnormal data type of the abnormal coded data may be information for characterizing the type of abnormal problem that occurs after the abnormal coded data is converted into a hexadecimal string. If the abnormal data type of the abnormal coded data is "TRUNCATE", it indicates that the abnormal problem that occurs after the abnormal coded data is converted into a hexadecimal string is that the converted hexadecimal string exceeds the upper limit of the length. If the abnormal data type of the abnormal coded data is "CONVERT_LOSSY", it indicates that the abnormal problem that occurs after the abnormal coded data is converted into a hexadecimal string is that garbled characters appear in the converted hexadecimal string. The data of the abnormal coded data may be a hexadecimal string obtained by reading the first 30 characters of the abnormal coded data in hexadecimal. The abnormal coded data, the stock data table where the abnormal coded data is located, and the hexadecimal string obtained after the abnormal coded data is converted may be analyzed and counted to determine the field type, field length, number of characters, number of original bytes, number of converted bytes, number of illegal characters, abnormal data type, and data of the abnormal coded data.

[0050] Optionally, for each detected abnormal coded data containing illegal characters in the stock data table, the stock data table name, identifier and field name of the abnormal coded data can be extracted from the stock data table where the abnormal coded data is located, so as to determine the stock data table name, identifier and field name of the abnormal coded data, and then the stock data table name, identifier and field name of the abnormal coded data are determined as the data information of the abnormal coded data, and the data information of the abnormal coded data is output to an abnormal data file. The abnormal data file can be a file preset for storing data information of the abnormal coded data.

[0051] Step 103, according to the abnormal data file, the normal coded data that does not contain illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormal coded data that contains illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal.

[0052] Optionally, after outputting the data information of all detected abnormal coded data to the abnormal data file, according to the abnormal data file, the normal coded data that does not contain illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormal coded data that contains illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal.

[0053] Optionally, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, including: starting from the first stock data table, each stock data table is transcoded and imported into the corresponding copy data table in turn, until the stock data table in the last position is processed. The normal coded data that does not contain illegal characters in each stock data table is transcoded and imported into the corresponding copy data table, including: performing the following operations for each stock data table: detecting whether the data information of each data in the stock data table is included in the abnormal data file, determining the data whose data information is not included in the abnormal data file as the normal coded data that does not contain illegal characters; transcoding each normal coded data that does not contain illegal characters, converting the character set of each normal coded data that does not contain illegal characters from the character set used by the database to be migrated into an intermediate character set; writing the transcoded normal coded data that does not contain illegal characters into the copy data table of the stock data table where the normal coded data that does not contain illegal characters is located. The data information that is not included in the abnormal data file is the normal encoded data that does not contain illegal characters.

[0054] Optionally, each item of normally encoded data that does not contain illegal characters can be transcoded, and the normally encoded data can be converted from data encoded according to the encoding rules of the character set used by the database to be migrated to data encoded according to the encoding rules of the intermediate character set, thereby converting the character set of the normally encoded data from the character set used by the database to be migrated to the intermediate character set. The intermediate character set can be a character set that is pre-set to be different from the character set used by the database to be migrated. Exemplarily, the intermediate character set is the UTF-8 character set. Each item of normally encoded data that does not contain illegal characters can be read through the Java Database Connection (JDBC) method of the non-domestic database, and the normally encoded data can be transcoded, and the normally encoded data can be converted from data encoded according to the encoding rules of the character set used by the database to be migrated to data encoded according to the encoding rules of the 16-bit Unicode Transformation Format (UTF-16) character set, thereby converting the character set of the normally encoded data from the character set used by the database to be migrated to the UTF-16 character set. Then, the JDBC method of the domestic database can be used to read the character set and convert it into normal encoded data that does not contain illegal characters in each item of the UTF-16 character set, transcode the normal encoded data, and convert the normal encoded data from data encoded according to the encoding rules of the UTF-16 character set into data encoded according to the encoding rules of the UTF-8 character set, thereby converting the character set of the normal encoded data from the UTF-16 character set to the UTF-8 character set.

[0055] Optionally, writing each item of the transcoded normal coded data that does not contain illegal characters into a copy data table of the stock data table where each item of the normal coded data that does not contain illegal characters is located, including: performing the following operations for each item of the normal coded data that does not contain illegal characters: determining the copy data table of the stock data table where the normal coded data in the target database is located according to the stock data table name of the normal coded data; wherein the data table in the target database with the same name as the stock data table name of the normal coded data is the copy data table of the stock data table where the normal coded data in the target database is located; and writing the transcoded normal coded data into the copy data table of the stock data table where the normal coded data is located according to the identifier and field name of the normal coded data, so that the position of the transcoded normal coded data in the copy data table of the stock data table where the normal coded data is located is the same as the position of the normal coded data in the stock data table where the normal coded data is located. The data table in the target database with the same name as the stock data table name of the normal coded data is determined as the copy data table of the stock data table where the normal coded data in the target database is located. The identifier and field name of the normal coded data can be used to identify the position of the normal coded data in the stock data table where the normal coded data is located. After determining the copy data table of the stock data table where the normally encoded data is located in the target database, the transcoded normally encoded data can be written to the copy data table of the stock data table where the normally encoded data is located according to the identifier and field name of the normally encoded data, so that the position of the transcoded normally encoded data in the copy data table of the stock data table where the normally encoded data is located is the same as the position of the normal encoded data in the stock data table where the normally encoded data is located.

[0056] Optionally, the abnormal coded data containing illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal, including: reading the data information of each abnormal coded data in the abnormal data file; reading each abnormal coded data in hexadecimal according to the data information of each abnormal coded data, and writing the read each abnormal coded data into the copy data table of the stock data table where each abnormal coded data is located. Under normal circumstances, when the abnormal coded data containing illegal characters is imported into the corresponding copy data table in the target database in hexadecimal, there will be no garbled code problem and import failure error problem.

[0057] In this way, based on the data information of each abnormal coded data in the abnormal data file, the abnormal coded data containing illegal characters in each detected inventory data table can be automatically imported into the target database in hexadecimal, avoiding garbled characters and import failure errors.

[0058] Optionally, the data information of each abnormal coded data in the abnormal data file can be read to obtain the stock data table name, identifier, field name and other information of each abnormal coded data. According to the data information of each abnormal coded data, each abnormal coded data is read in hexadecimal, and each abnormal coded data read is written into the copy data table of the stock data table where each abnormal coded data is located, including: performing the following operations for each acquired data information of abnormal coded data: according to the stock data table name, identifier and field name in the data information, determining the stock data table for storing the abnormal coded data belonging to the data information, and the position of the abnormal coded data belonging to the data information in the stock data table; reading the abnormal coded data belonging to the data information in hexadecimal, thereby converting the abnormal coded data belonging to the data information into a hexadecimal string; writing the abnormal coded data belonging to the data information converted into a hexadecimal string into the copy data table of the stock data table for storing the abnormal coded data belonging to the data information.

[0059] Optionally, writing the abnormal coded data belonging to the data information converted into a hexadecimal string into a copy data table of a stock data table used to store the abnormal coded data belonging to the data information includes: determining the copy data table of the stock data table in the target database used to store the abnormal coded data belonging to the data information according to the stock data table name in the data information; wherein a data table in the target database having the same name as the stock data table name in the data information is the copy data table of the stock data table in the target database used to store the abnormal coded data belonging to the data information; and writing the abnormal coded data belonging to the data information converted into a hexadecimal string into the copy data table of the stock data table used to store the abnormal coded data belonging to the data information according to the identifier and field name in the data information, so that the position of the abnormal coded data belonging to the data information converted into a hexadecimal string in the copy data table of the stock data table used to store the abnormal coded data belonging to the data information is the same as the position of the abnormal coded data belonging to the data information in the stock data table.

[0060] Optionally, a data table in the target database having the same name as the stock data table in the data information can be determined as a copy data table of the stock data table in the target database for storing abnormally coded data belonging to the data information. The identifier and field name in the data information can be used to identify the position of the abnormally coded data belonging to the data information in the stock data table. After determining the copy data table of the stock data table in the target database for storing abnormally coded data belonging to the data information, the abnormally coded data belonging to the data information converted into a hexadecimal string can be written into the copy data table of the stock data table for storing abnormally coded data belonging to the data information according to the identifier and field name in the data information through the character conversion function cast_to_varchar2, so that the position of the abnormally coded data belonging to the data information converted into a hexadecimal string in the copy data table of the stock data table for storing abnormally coded data belonging to the data information is the same as the position of the abnormally coded data belonging to the data information in the stock data table.

[0061] Step 104: Detect whether the data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table.

[0062] Optionally, an illegal character detection tool may be used to read each data item in each incremental data table in the database to be migrated in hexadecimal format and detect whether each data item contains illegal characters.

[0063] Optionally, each incremental data table in the database to be migrated is arranged in a target order. The target order may be the order of arrangement of each incremental data table specified by the technician. Starting from the incremental data table ranked first, the illegal character detection tool may be used in turn to read each data in each incremental data table in hexadecimal and detect whether each data contains illegal characters, until the processing of the last incremental data table is completed. For each incremental data table, the illegal character detection tool may be used to perform the following operations: read each data in the incremental data table in hexadecimal, thereby converting each data in the incremental data table into a hexadecimal string; detect whether each hexadecimal string contains illegal characters; for each hexadecimal string containing illegal characters detected, determine that the data to which the hexadecimal string belongs contains illegal characters, and the data is abnormally encoded data containing illegal characters; for each hexadecimal string that does not contain illegal characters detected, determine that the data to which the hexadecimal string belongs does not contain illegal characters, and the data is normal encoded data that does not contain illegal characters.

[0064] Optionally, an abnormal incremental data table may refer to an incremental data table in each incremental data table that stores abnormal coded data containing illegal characters. An abnormal incremental file may be a file for storing the name of an abnormal incremental data table. A normal incremental data table may refer to an incremental data table in each incremental data table that does not store abnormal coded data containing illegal characters. A normal incremental file may be a file for storing the name of a normal incremental data table. An incremental data table that contains abnormal coded data containing illegal characters is configured as an abnormal incremental data table, and an incremental data table that does not contain abnormal coded data containing illegal characters is configured as a normal incremental data table, including: for each incremental data table that contains abnormal coded data containing illegal characters, the name of the incremental data table is written into the abnormal incremental file; for each incremental data table that does not contain abnormal coded data containing illegal characters, the name of the incremental data table is written into the normal incremental file.

[0065] Step 105: Transcode the data in each normal incremental data table and import it into the corresponding replica data table, and import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal.

[0066] Optionally, the data in each normal incremental data table is transcoded and imported into the corresponding replica data table, and the data in each abnormal incremental data table is imported into the corresponding replica data table in hexadecimal, including: for each normal incremental data table, transcoding each data in the normal incremental data table, converting the character set of each data into an intermediate character set, arranging the transcoded data in chronological order of occurrence of each data, and obtaining data transactions of the normal incremental data table; for each abnormal incremental data table, reading each data in the abnormal incremental data table in hexadecimal, arranging the read data in chronological order of occurrence of each data, and obtaining data transactions of the abnormal incremental data table; writing the data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table to the target disk; generating a data queue of each normal incremental data table according to the data transactions of each normal incremental data table; generating a data queue of each abnormal incremental data table according to the data transactions of each abnormal incremental data table; writing the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table to the corresponding replica data table.

[0067] Therefore, based on the illegal character detection results of the incremental data table, the data in each incremental data table can be imported into the target database, the data in each incremental data table that does not contain abnormally encoded data containing illegal characters can be transcoded and imported into the corresponding copy data table, and the data in each incremental data table that contains abnormally encoded data containing illegal characters can be imported into the corresponding copy data table in hexadecimal, avoiding garbled characters and import failure errors.

[0068] Optionally, for each normal incremental data table, each data in the normal incremental data table can be read, and each data in the normal incremental data table can be transcoded, and each data can be converted from data encoded according to the encoding rules of the character set used by the database to be migrated to data encoded according to the encoding rules of the intermediate character set, so as to convert the character set of each data from the character set used by the database to be migrated to the intermediate character set, and then the transcoded data can be arranged in the order of the occurrence time of each data to form a data sequence. The formed data sequence is the data transaction of the normal incremental data table. The occurrence time of the data can refer to the time when the data is generated. The occurrence time of each data is stored in the log of the database to be migrated. The occurrence time of each data can be read from the log of the database to be migrated, and the transcoded data can be arranged in the order of the occurrence time of each data to form a data sequence. The earliest occurrence time is arranged first, and the latest occurrence time is arranged last. The occurrence time of each data can be read from the log of the database to be migrated through LogMiner, an analysis tool of the database to be migrated.

[0069] Optionally, for each abnormal incremental data table, each data in the abnormal incremental data table can be read in hexadecimal, so as to convert each data in the abnormal incremental data table into a hexadecimal string, and then the data converted into the hexadecimal string are arranged in the order of occurrence time of each data to form a data sequence. The formed data sequence is the data transaction of the abnormal incremental data table. The earliest occurrence time is arranged first, and the latest occurrence time is arranged last.

[0070] Optionally, the target disk may be a disk for storing data of the database to be migrated. The data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table are written to the target disk respectively.

[0071] Optionally, the data queue of the normal incremental data table may refer to a queue for storing data transactions of the normal incremental data table. According to the data transactions of each normal incremental data table, the data queue of each normal incremental data table is generated, including: performing the following operations for each data transaction of the normal incremental data table: creating a queue for storing data transactions of the normal incremental data table; writing the data transactions of the normal incremental data table to the created queue to form a data queue of the normal incremental data table.

[0072] Optionally, the data queue of the abnormal incremental data table may refer to a queue for storing data transactions of the abnormal incremental data table. According to the data transactions of each abnormal incremental data table, the data queues of each abnormal incremental data table are generated, including: performing the following operations for each data transaction of the abnormal incremental data table: creating a queue for storing data transactions of the abnormal incremental data table; writing the data transactions of the abnormal incremental data table to the created queue to form the data queue of the abnormal incremental data table.

[0073] Optionally, the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table are written into the corresponding replica data table, including: determining the data thread corresponding to each data queue; reading the data in each data queue in parallel through the data thread corresponding to each data queue, and writing the read data into the replica data table of the data table to which each data queue belongs.

[0074] Therefore, based on multiple threads, the data in each incremental data table can be imported into the target database in parallel, thereby improving the data migration efficiency.

[0075] Optionally, the data thread may be a thread for writing data in the queue to the data table. Determining the data thread corresponding to each data queue includes: allocating a data thread to each data queue of each normal incremental data table and each data queue of each abnormal incremental data table, thereby determining the data thread corresponding to each data queue.

[0076] Optionally, the replica data table of the data table to which each data queue belongs refers to a replica data table of a normal incremental data table or an abnormal incremental data table to which each data queue belongs in the target database. The data in each data queue can be read in parallel through the data threads corresponding to each data queue, and the read data can be written to the replica data table of the data table to which each data queue belongs, so that the transcoded data in each normal incremental data table can be written to the corresponding replica data table in the target database, and the data converted into hexadecimal strings in each abnormal incremental data table can be written to the corresponding replica data table in the target database.

[0077] Optionally, the data format of the transcoded normal coded data is data encoded according to the encoding rules of the intermediate character set. After the normal coded data that does not contain illegal characters is transcoded and imported into the corresponding copy data table, the target database can determine that the transcoded normal coded data is normal coded data that does not contain illegal characters encoded according to the encoding rules of the intermediate character set based on the data format of the transcoded normal coded data. If the character library of the target database is not the intermediate character set, the target database can transcode each item of normal coded data that does not contain illegal characters encoded according to the encoding rules of the intermediate character set, and convert the normal coded data that does not contain illegal characters encoded according to the encoding rules of the intermediate character set from data encoded according to the encoding rules of the intermediate character set to data encoded according to the encoding rules of the character set used by the target database, thereby converting the character set of the normal coded data that does not contain illegal characters from the intermediate character set to the character set used by the target database.

[0078] Optionally, the data format of the abnormal coded data converted to a hexadecimal string is a hexadecimal string. After the abnormal coded data containing illegal characters is imported into the corresponding replica data table in hexadecimal, the target database can determine that the data is abnormal coded data containing illegal characters converted to a hexadecimal string based on the data format of the data. The target database can perform transcoding for each abnormal coded data containing illegal characters converted to a hexadecimal string, and convert the abnormal coded data containing illegal characters converted to a hexadecimal string into data encoded according to the encoding rules of the character set used by the target database.

[0079] The technical solution of the embodiment of the present invention is to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in the target database corresponding to the database to be migrated; then detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each detected abnormal coded data containing illegal characters to an abnormal data file; according to the abnormal data file, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, and the abnormal coded data containing illegal characters in each stock data table detected is imported into the corresponding copy data table in hexadecimal; then detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormal coded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormal coded data containing illegal characters as a normal incremental data table; finally, each normal incremental The data in the data table is transcoded and imported into the corresponding copy data table, and the data in each abnormal incremental data table is imported into the corresponding copy data table in hexadecimal, which solves the problem that the data migration solution in the related technology cannot process illegal characters in the non-domestic database, resulting in the import of data containing garbled characters into the domestic database or causing the domestic database to fail to import and report errors, and the correctness of data migration cannot be guaranteed. Illegal character detection can be automatically performed on the existing data and incremental data in the non-domestic database. Based on the illegal character detection results of the existing data, the data containing illegal characters in the existing data is imported into the domestic database in hexadecimal. Based on the illegal character detection results of the incremental data, the data in each incremental data table with abnormally encoded data containing illegal characters is imported into the domestic database in hexadecimal, avoiding garbled characters and import failure errors. Illegal characters in non-domestic databases can be processed to avoid importing garbled characters into the domestic database or causing the domestic database to fail to import and report errors, thereby ensuring the correctness of data migration.

[0080] Embodiment 2

[0081] Figure 2 This is a flow chart of a data migration method provided by Embodiment 2 of the present invention. This embodiment of the present invention can be combined with each optional solution in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0082] Step 201: determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in a target database corresponding to the database to be migrated.

[0083] Step 202: Detect whether each data item in each stock data table in the database to be migrated contains illegal characters, and output data information of each detected abnormal coded data item containing illegal characters to an abnormal data file.

[0084] Step 203: According to the abnormal data file, the normal coded data that does not contain illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormal coded data that contains illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal.

[0085] Step 204: Detect whether the data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table.

[0086] Step 205: For each normal incremental data table, transcode each data in the normal incremental data table, convert the character set of each data into an intermediate character set, arrange the transcoded data in chronological order of occurrence of each data, and obtain the data transaction of the normal incremental data table.

[0087] Step 206: For each abnormal incremental data table, read each data in the abnormal incremental data table in hexadecimal, arrange the read data in chronological order according to the occurrence time of each data, and obtain the data transaction of the abnormal incremental data table.

[0088] Step 207: Write the data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table to the target disk.

[0089] Step 208: Generate a data queue for each normal incremental data table according to the data transaction of each normal incremental data table.

[0090] Step 209: Generate a data queue for each abnormal incremental data table according to the data transaction of each abnormal incremental data table.

[0091] Step 210: Write the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table into the corresponding replica data table.

[0092] The technical solution of the embodiment of the present invention can import the data in each incremental data table into the target database based on the illegal character detection result of the incremental data table, transcode the data in each incremental data table that does not contain abnormally encoded data containing illegal characters and import it into the corresponding copy data table, and import the data in each incremental data table that contains abnormally encoded data containing illegal characters in hexadecimal into the corresponding copy data table, so as to avoid garbled characters and import failure error problems.

[0093] Embodiment 3

[0094] Figure 3 Schematic diagram of a data migration device provided in Embodiment 3 of the present invention. The device may be configured in an electronic device. Figure 3 As shown, the device includes: a data table determination module 301, an existing data detection module 302, an existing data migration module 303, an incremental data detection module 304 and an incremental data migration module 305.

[0095] Among them, the data table determination module 301 is used to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in the target database corresponding to the database to be migrated; the stock data detection module 302 is used to detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to the abnormal data file; the stock data migration module 303 is used to transcode the normal coded data that does not contain illegal characters in each stock data table detected according to the abnormal data file and import it into the corresponding copy data table , import the abnormal coded data containing illegal characters in each stock data table detected into the corresponding replica data table in hexadecimal; the incremental data detection module 304 is used to detect whether the data in each incremental data table in the database to be migrated contains illegal characters, and configure the incremental data table with abnormal coded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormal coded data containing illegal characters as a normal incremental data table; the incremental data migration module 305 is used to transcode the data in each normal incremental data table and import it into the corresponding replica data table, and import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal.

[0096] The technical solution of the embodiment of the present invention is to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in the target database corresponding to the database to be migrated; then detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each detected abnormal coded data containing illegal characters to an abnormal data file; according to the abnormal data file, the normal coded data that does not contain illegal characters in each stock data table detected is transcoded and imported into the corresponding copy data table, and the abnormal coded data containing illegal characters in each stock data table detected is imported into the corresponding copy data table in hexadecimal; then detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormal coded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormal coded data containing illegal characters as a normal incremental data table; finally, each normal incremental The data in the data table is transcoded and imported into the corresponding copy data table, and the data in each abnormal incremental data table is imported into the corresponding copy data table in hexadecimal, which solves the problem that the data migration solution in the related technology cannot process illegal characters in the non-domestic database, resulting in the import of data containing garbled characters into the domestic database or causing the domestic database to fail to import and report errors, and the correctness of data migration cannot be guaranteed. Illegal character detection can be automatically performed on the existing data and incremental data in the non-domestic database. Based on the illegal character detection results of the existing data, the data containing illegal characters in the existing data is imported into the domestic database in hexadecimal. Based on the illegal character detection results of the incremental data, the data in each incremental data table with abnormally encoded data containing illegal characters is imported into the domestic database in hexadecimal, avoiding garbled characters and import failure errors. Illegal characters in non-domestic databases can be processed to avoid importing garbled characters into the domestic database or causing the domestic database to fail to import and report errors, thereby ensuring the correctness of data migration.

[0097] In an optional implementation of an embodiment of the present invention, optionally, when the stock data detection module 302 performs the operation of detecting whether each data in each stock data table in the database to be migrated contains illegal characters, it is specifically used to: read each data in each stock data table in the database to be migrated in hexadecimal through an illegal character detection tool and detect whether each data contains illegal characters.

[0098] In an optional implementation of an embodiment of the present invention, optionally, when the stock data detection module 302 performs the operation of outputting the data information of each detected abnormal coded data containing illegal characters to an abnormal data file, it is specifically used to: for each detected abnormal coded data containing illegal characters in the stock data table, determine the stock data table name, identifier and field name of the abnormal coded data as the data information of the abnormal coded data, and output the data information of the abnormal coded data to the abnormal data file.

[0099] In an optional implementation of an embodiment of the present invention, optionally, when the stock data migration module 303 performs the operation of importing the detected abnormal coded data containing illegal characters in each stock data table into the corresponding copy data table in hexadecimal, it is specifically used to: read the data information of each abnormal coded data in the abnormal data file; read each abnormal coded data in hexadecimal according to the data information of each abnormal coded data, and write the read each abnormal coded data into the copy data table of the stock data table where the abnormal coded data is located.

[0100] In an optional implementation of an embodiment of the present invention, optionally, the incremental data migration module 305 is specifically used to: for each normal incremental data table, transcode each data in the normal incremental data table, convert the character set of each data into an intermediate character set, arrange the transcoded data in chronological order of occurrence of each data, and obtain data transactions of the normal incremental data table; for each abnormal incremental data table, read each data in the abnormal incremental data table in hexadecimal, arrange the read data in chronological order of occurrence of each data, and obtain data transactions of the abnormal incremental data table; write the data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table to the target disk; generate a data queue for each normal incremental data table according to the data transactions of each normal incremental data table; generate a data queue for each abnormal incremental data table according to the data transactions of each abnormal incremental data table; write the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table to the corresponding copy data table.

[0101] In an optional implementation of an embodiment of the present invention, optionally, when the incremental data migration module 305 executes the operation of writing the data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table into the corresponding copy data table, it is specifically used to: determine the data thread corresponding to each data queue; read the data in each data queue in parallel through the data thread corresponding to each data queue, and write the read data into the copy data table of the data table to which each data queue belongs.

[0102] The data migration device provided in the embodiment of the present invention can execute the data migration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0103] Embodiment 4

[0104] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement the data migration method of an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0105] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0107] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data migration method.

[0108] In some embodiments, the data migration method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the data migration method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the data migration method in any other appropriate manner (e.g., by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or electronic device.

[0111] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein may be implemented on a heterogeneous hardware accelerator having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the heterogeneous hardware accelerator. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data electronic device), or a computing system that includes middleware components (e.g., an application electronic device), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] The computing system may include a client and an electronic device. The client and the electronic device are generally remote from each other and usually interact through a communication network. The relationship between the client and the electronic device is generated by computer programs running on corresponding computers and having a client-electronic device relationship with each other. The electronic device may be a cloud electronic device, also known as a cloud computing electronic device or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data migration method, characterized in that: include: Determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in a target database corresponding to the database to be migrated; Detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to an abnormal data file; According to the abnormal data file, the normal coded data without illegal characters in each detected stock data table is transcoded and imported into the corresponding copy data table, and the abnormal coded data containing illegal characters in each detected stock data table is imported into the corresponding copy data table in hexadecimal format; Detect whether each data in each incremental data table in the database to be migrated contains illegal characters, configure the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configure the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table; The data in each normal incremental data table is transcoded and imported into the corresponding replica data table, and the data in each abnormal incremental data table is imported into the corresponding replica data table in hexadecimal format.

2. The data migration method according to claim 1, characterized in that: Check whether the data in each stock data table in the database to be migrated contains illegal characters, including: Use the illegal character detection tool to read the data in each stock data table in the database to be migrated in hexadecimal format and detect whether the data contains illegal characters.

3. The data migration method according to claim 1, characterized in that: Output the data information of each abnormal coded data containing illegal characters detected to the abnormal data file, including: For each abnormal coded data containing illegal characters detected in the stock data table, the stock data table name, identifier and field name of the abnormal coded data are determined as data information of the abnormal coded data, and the data information of the abnormal coded data is output to the abnormal data file.

4. The data migration method according to claim 1, characterized in that: Import the abnormally encoded data containing illegal characters in each inventory data table detected into the corresponding copy data table in hexadecimal format, including: Read the data information of each abnormal coded data in the abnormal data file; According to the data information of each abnormal coding data, each abnormal coding data is read in hexadecimal, and the read each abnormal coding data is written into the copy data table of the stock data table where the each abnormal coding data is located.

5. The data migration method according to claim 1, characterized in that: Transcode the data in each normal incremental data table and import it into the corresponding replica data table. Import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal format, including: For each normal incremental data table, transcode each data in the normal incremental data table, convert the character set of each data into an intermediate character set, arrange the transcoded data in the order of occurrence time of each data, and obtain the data transaction of the normal incremental data table; For each abnormal incremental data table, read each data in the abnormal incremental data table in hexadecimal, arrange the read data in the order of occurrence time of each data, and obtain the data transaction of the abnormal incremental data table; Write the data transactions of each normal incremental data table and the data transactions of each abnormal incremental data table to the target disk; Generate data queues for each normal incremental data table based on data transactions of each normal incremental data table; Generate data queues for each abnormal incremental data table based on data transactions of each abnormal incremental data table; The data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table are written into the corresponding replica data table.

6. The data migration method according to claim 5, characterized in that: The data in the data queue of each normal incremental data table and the data in the data queue of each abnormal incremental data table are written into the corresponding replica data table, including: Determine the data threads corresponding to each data queue; The data in each data queue is read in parallel through the data threads corresponding to each data queue, and the read data is written into the replica data table of the data table to which each data queue belongs.

7. A data migration device, characterized in that: include: A data table determination module, used to determine each stock data table and each incremental data table in the database to be migrated, and create a copy data table of each stock data table and each incremental data table in a target database corresponding to the database to be migrated; The stock data detection module is used to detect whether each data in each stock data table in the database to be migrated contains illegal characters, and output the data information of each abnormal coded data containing illegal characters detected to the abnormal data file; The stock data migration module is used to, according to the abnormal data file, transcode the normal coded data that does not contain illegal characters in each detected stock data table and import it into the corresponding copy data table, and import the abnormal coded data that contains illegal characters in each detected stock data table in hexadecimal into the corresponding copy data table; An incremental data detection module is used to detect whether each data in each incremental data table in the database to be migrated contains illegal characters, and configures the incremental data table with abnormally encoded data containing illegal characters as an abnormal incremental data table, and configures the incremental data table without abnormally encoded data containing illegal characters as a normal incremental data table; The incremental data migration module is used to transcode the data in each normal incremental data table and import it into the corresponding replica data table, and import the data in each abnormal incremental data table into the corresponding replica data table in hexadecimal.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data migration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data migration method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the data migration method according to any one of claims 1 to 6.

Images