Artificial Intelligence-Based Data Migration Method and Related Devices

Abstract

This application proposes an artificial intelligence-based data migration method, device, electronic device, and storage medium. The artificial intelligence-based data migration method includes: obtaining database information of a source database and a target database; configuring table structures of the source database and the target database based on the database information to obtain a table structure correspondence relationship; matching the source database and the target database based on the table structure correspondence relationship to obtain a statement change set; dividing access traffic information of the source database based on the statement change set and a clustering algorithm to obtain a synchronization time set; and synchronizing data in the source database to the target database based on the statement change set and the synchronization time set to complete data migration. This application can perform data migration in segments according to the access traffic of the source database, improving the efficiency of the normal operation of the system.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a data migration method, device, electronic device, and storage medium based on artificial intelligence. Background Art

[0002] With the development of information technology, the usage and exchange volume of data are both increasing rapidly. In order to improve performance and expand business, many companies inevitably need to migrate data in the database. When migrating data between databases, the existing technology usually first parses the table structure definition of the source database, parses it into the table structure supported by the target database, and generates a new table in the target database before migrating the data.

[0003] However, although the existing technology can achieve the migration of multiple databases, the data migration process is long and the migration efficiency is low, resulting in the source database being unable to be used during the migration process, thereby seriously affecting the normal operation efficiency of the system functions. Summary of the Invention

[0004] In view of the above, it is necessary to propose a data migration method and related devices based on artificial intelligence to solve the technical problem of how to improve the normal operation efficiency of the system during the database migration process. Among them, the related devices include a data migration device, an electronic device, and a storage medium based on artificial intelligence.

[0005] This application provides a data migration method based on artificial intelligence, including:

[0006] Obtain the database information of the source database and the target database;

[0007] Configure the table structures of the source database and the target database based on the database information to obtain the corresponding relationship of the table structures;

[0008] Judge whether the table structure information of the source database and the target database is consistent based on the corresponding relationship of the table structures to obtain the migration table structure;

[0009] Match the source database and the target database based on the migration table structure to obtain the statement change set;

[0010] Divide the access traffic information of the source database based on the statement change set and the clustering algorithm to obtain the synchronization time set;

[0011] Synchronize the data in the source database to the target database based on the statement change set and the synchronization time set to complete the data migration.

[0012] In this way, by obtaining the information of the source database to construct a table structure that conforms to the target database type, and matching the statement change information between the source database and the target database based on the table structure information as the data to be migrated, and combining the access traffic of the source database to segmentally migrate the data to be migrated during the period when the normal operation of the system is minimally affected, the efficiency of the normal operation of the system can be effectively improved.

[0013] In some embodiments, the obtaining of the database information of the source database and the target database includes:

[0014] Receiving, according to a web page, the database information of the source database input by a user, where the database information of the source database includes a connection name, a database type, a schema, a host IP, a database name, a port, a user name, and a password;

[0015] Receiving, according to a web page, the database information of the target database input by a user, where the database information of the target database includes a database table name, a synchronization method, a connection name, a database type, a host IP, a database name, a port, a user name, a password, and a target database table name.

[0016] In this way, during the process of database migration, there is no need to install a client, and the database migration can be conveniently completed only by configuring through an opened web page. At the same time, the user can determine the information of the source database and the target database through the web page, which can effectively improve the user's human-computer interaction experience and facilitate the user to flexibly configure the database information customarily.

[0017] In some embodiments, the configuring of the table structures of the source database and the target database based on the database information to obtain a corresponding relationship between the table structures includes:

[0018] Judging whether the types of the source database and the target database are the same based on the database information to obtain a judgment result, where the judgment result is the same and different;

[0019] If the judgment result is the same, then configure the corresponding relationship between the table structure of the source database and the table structure of the target database;

[0020] If the judgment result is different, then restructure the table structure of the source database into a format that conforms to the table structure of the target database, and then configure the corresponding relationship between the table structure of the source database and the table structure of the target database.

[0021] In this way, by judging whether the types of the source database and the target database are the same based on the database information, it can be ensured that the types of the source database and the target database are the same, thus providing the possibility for data migration in the subsequent process.

[0022] In some embodiments, matching the source database and the target database based on the table structure correspondence to obtain a statement change set includes:

[0023] Performing a structure analysis on each data table in the source database and the target database to obtain a source data table structure set and a target data table structure set;

[0024] Based on the table structure correspondence, comparing the table structure information of the corresponding tables in the source data table structure set and the target data table structure set one by one to obtain a statement change set.

[0025] In this way, the content differences between each table in the source database and the corresponding tables with the same table structure in the target database can be obtained through the structure analysis of the data table structure, thereby providing the possibility for accurately dividing the access traffic by using this content difference in the subsequent process.

[0026] In some embodiments, matching the source database and the target database based on the data table structure set to obtain a statement change set includes:

[0027] If the table structure information of the source data table structure set and the target data table structure set is consistent, the statement change set is an empty set;

[0028] If the table structure information of the source data table structure set and the target data table structure set is inconsistent, a structure modification script is generated, and the table structure information of the target data table structure set is modified based on the structure modification script.

[0029] In this way, a structure modification script can be automatically generated according to the differences in the table structure information of the source database and the target database to ensure that the table structures of the source database and the target database are consistent, which can effectively save human resources and improve the efficiency of data migration compared with the manual method.

[0030] In some embodiments, the step of generating a structure modification script if the table structure information of the source data table structure set and the target data table structure set is inconsistent includes:

[0031] Analyzing the table structure information of the source data table structure set and the target data table structure set to obtain difference information;

[0032] Based on the difference information, splicing SQL statements to generate standard data manipulation statements;

[0033] Generating a structure modification script based on the standard data manipulation statements.

[0034] In this way, by obtaining the difference information to splice the SQL statement, users can flexibly add, delete, or modify the content that needs to be changed in the database according to their needs, realizing the basic operations on the database. At the same time, the standard data manipulation statements can be used as programming statements for accessing the objects and data in the database to generate corresponding structure modification scripts.

[0035] In some embodiments, dividing the access traffic information of the source database based on the statement change set and the clustering algorithm to obtain the synchronization time set includes:

[0036] Using a traffic monitoring tool to obtain the access traffic information of the source database;

[0037] Using the K-means clustering algorithm to divide the access traffic information, where the value of K is specified and obtained by the statement change set;

[0038] Taking the time period corresponding to each category of the divided access traffic information as the synchronization time period, and all the synchronization time periods constitute the synchronization time set.

[0039] In this way, the data to be migrated can be segmented and migrated within the time period that does not affect the normal operation of the system as much as possible according to the access traffic of the source database, thereby improving the efficiency of the normal operation of the system.

[0040] The embodiment of the present application also provides an artificial intelligence-based data migration device, including:

[0041] An acquisition unit, configured to acquire the database information of the source database and the target database;

[0042] A configuration unit, configured to configure the table structures of the source database and the target database based on the database information to obtain the table structure correspondence;

[0043] A matching unit, configured to match the source database and the target database based on the table structure correspondence to obtain a statement change set;

[0044] A division unit, configured to divide the access traffic information of the source database based on the statement change set and the clustering algorithm to obtain a synchronization time set;

[0045] A synchronization unit, configured to synchronize the data in the source database to the target database based on the statement change set and the synchronization time set to complete the data migration.

[0046] The embodiment of the present application also provides an electronic device, including:

[0047] A memory, storing at least one instruction;

[0048] A processor that executes instructions stored in the memory to implement the above-mentioned artificial intelligence-based data migration method.

[0049] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the above-mentioned artificial intelligence-based data migration method. Description of the Drawings

[0050] Figure 1 is a flowchart of a preferred embodiment of the artificial intelligence-based data migration method involved in the present application.

[0051] Figure 2 is a flowchart of a preferred embodiment of obtaining database information of a source database and a target database involved in the present application.

[0052] Figure 3 is a flowchart of a preferred embodiment of configuring table structures of a source database and a target database based on database information to obtain a corresponding relationship of table structures involved in the present application.

[0053] Figure 4 is a functional module diagram of a preferred embodiment of the artificial intelligence-based data migration device involved in the present application.

[0054] Figure 5 is a schematic structural diagram of an electronic device of a preferred embodiment of the artificial intelligence-based data migration method involved in the present application. Detailed Embodiments

[0055] In order to more clearly understand the purpose, features, and advantages of the present application, the present application will be described in detail below with reference to the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other. Many specific details are set forth in the following description in order to fully understand the present application. The described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0056] In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality" means two or more, unless otherwise specifically defined.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0058] An embodiment of this application provides an artificial intelligence-based data migration method, which can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0059] An electronic device can be any electronic product that can perform human-computer interaction with a user. For example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an Internet Protocol Television (IPTV), a smart wearable device, etc.

[0060] The electronic device may also include a network device and / or a user device. Among them, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0061] The network where the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0062] As Figure 1 shown, it is a flowchart of a preferred embodiment of the artificial intelligence-based data migration method of this application. According to different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0063] S10, obtain the database information of the source database and the target database.

[0064] As Figure 2As shown, in an alternative embodiment, the source database is the source library for data synchronization. The source database can be various databases compliant with standard SQL. The target database is the target library for data synchronization, and the target database can be various databases compliant with standard SQL.

[0065] In an alternative embodiment, the obtaining of the database information of the source database and the target database includes:

[0066] S101. According to the web page, receive the database information of the source database input by the user. The database information of the source database includes connection name, database type, schema, host IP, database name, port, username, password, and other contents.

[0067] In this alternative embodiment, when the user needs to perform database migration, the user can open a web page and input configuration information on the web page to achieve the purpose of database migration based on the web. First, the web page opened by the user can include the information of the source database that the user needs to input. The information of the source database that needs to be input can specifically include: connection name, database type, schema, host IP, database name, port, username, password, and other contents.

[0068] In this alternative embodiment, after inputting the information of the source database, the user can determine some or all of the data tables and some data fields in the data tables in the source database as the data to be migrated according to the requirements. Among them, the methods for determining the data information to be migrated can include: selecting by cursor, inputting the specific positions of the data tables or data fields to be migrated in the database to be migrated, and directly inputting the data content to be migrated.

[0069] S102. According to the web page, receive the database information of the target database input by the user. The database information of the target database includes database table name, synchronization method, connection name, database type, host IP, database name, port, username, password, and target library table name, and other contents.

[0070] In this alternative embodiment, after determining the information of the source database, the information of the target database can be input in another web page. In this web page, the information of the target database can include: database table name, synchronization method, connection name, database type, host IP, database name, port, username, password, and target library table name, and other contents. Among them, the information of the source database and the information of the target database are not completely the same. The information of the target database depends on the specific type and configuration of the target database.

[0071] In this way, during the process of database migration, there is no need to install a client. It is only necessary to configure through an opened web page to conveniently complete the database migration. At the same time, the user can determine the information of the source database and the target database through the web page, which can effectively improve the user's human-computer interaction experience and facilitate the user to customize and flexibly configure the database information.

[0072] S11. Configure the table structures of the source database and the target database based on the database information to obtain the corresponding relationship of the table structures.

[0073] As Figure 3 shown, in an optional embodiment, a database generally stores data in a library table structure. The library table structure refers to the table structure of the database. The table structure is composed of three parts: the table name, the fields in the table, and the records of the table. The table structure describes the framework of a database, defines the number of fields that make up a database, the name of each field, the data type of each field, and the length of each field, etc. Before establishing a database, it is necessary to pre-design the data source table structure and store the relevant information of the table structure in the metadata of the database.

[0074] In this optional embodiment, metadata is data used to describe data (data about data), mainly information describing data properties (property), and supports functions such as indicating storage location, historical data, resource search, file records, etc. In this embodiment, the metadata is saved in an Excel file and includes field information such as table name, group name, library name, information of the source table structure, incremental timestamp, etc.

[0075] In an optional embodiment, configuring the table structures of the source database and the target database based on the database information to obtain the corresponding relationship of the table structures includes:

[0076] S111. Judge whether the types of the source database and the target database are the same based on the database information to obtain a judgment result, and the judgment result is the same and different.

[0077] In this optional embodiment, based on the database information, the database environment information of the system operation can be obtained. Based on the database environment information, the database type can be obtained. The database environment information is saved in a configuration file and is obtained when creating a database connection.

[0078] In an optional embodiment, the corresponding relationship of the table structures includes information such as the corresponding relationship between data tables and the mapping relationship between data table fields. The database types of the source database and the target database may not be the same. For example, the source database is oracle and the target database is mysql.

[0079] In this optional embodiment, the corresponding relationships of the table structures include, but are not limited to, one-to-one and one-to-many. Among them, one-to-one means "vertically splitting" the data table, that is, one record in Table A corresponds to one record in Table B; one-to-many means one record in Table A corresponds to multiple records in Table B. For example, in a class table and a student table, one class can correspond to multiple students.

[0080] S112. If the judgment result is consistent, configure the corresponding relationship between the table structure of the source database and the table structure of the target database.

[0081] In this optional embodiment, if the judgment result is consistent, the table structures of the source database and the target database are associated through an association program, thereby completing the configuration of the corresponding relationship of the table structures. The association program is a preset script program, which is used to find the corresponding table structure information in the target database based on information such as the table name and table fields in the table structure information of the source database and perform the association.

[0082] S113. If the judgment result is inconsistent, reorganize the table structure of the source database into a format that conforms to the table structure of the target database, and then configure the corresponding relationship between the table structure of the source database and the table structure of the target database.

[0083] In this optional embodiment, if the judgment result is inconsistent, a reorganization program is used to recombine the table structure of the source database according to the table structure of the target database to obtain a format that conforms to the table structure of the target database, and then the table structures of the source database and the target database are associated through an association program, thereby completing the configuration of the corresponding relationship of the table structures. The reorganization program is a preset script program, which is used to recombine the table structure of the source database according to information such as the table name and table fields in the table structure information of the target database to obtain a format that conforms to the table structure of the target database.

[0084] In this way, by judging whether the types of the source database and the target database are consistent through the database information, it can be ensured that the types of the source database and the target database are the same, thereby providing the possibility for data migration in the subsequent process.

[0085] S12. Match the source database and the target database based on the table structure correspondence to obtain a statement change set.

[0086] In an optional embodiment, matching the source database and the target database based on the table structure correspondence to obtain a statement change set includes:

[0087] S121. Analyze the structure of each data table in the source database and the target database to obtain a source data table structure set and a target data table structure set.

[0088] In this optional embodiment, analyzing the structure of each data table in the source database and the target database means obtaining the structure information of each data table, that is, obtaining the table name, fields in the table, and records of each table. Finally, the structure information of each table constitutes the data table structure set of the database.

[0089] S122. Based on the table structure correspondence, compare one by one whether the table structure information of the corresponding tables in the source data table structure set and the target data table structure set is consistent to obtain a statement change set.

[0090] In this optional embodiment, by comparing whether there are differences in the field information of the source database table structure and the corresponding table structure of the target database, it is determined whether the source data table structure needs to change during data migration. That is, compared with the target database table structure, whether fields are added, deleted, or the attributes of the fields (field name, field length, and data type of the field, etc.) are modified, and the finally matched different field information is used as the statement change set.

[0091] In an optional embodiment, the step of comparing one by one whether the table structure information of the corresponding tables in the source data table structure set and the target data table structure set is consistent based on the table structure correspondence to obtain a statement change set includes: if the table structure information of the source data table structure set and the target data table structure set is consistent, then the statement change set is an empty set; if the table structure information of the source data table structure set and the target data table structure set is inconsistent, then generate a structure modification script, and modify the table structure information of the target data table structure set based on the structure modification script.

[0092] In an optional embodiment, the step of generating a structure modification script when the table structure information of the source data table structure set and the target data table structure set is inconsistent includes: analyzing the table structure information of the source data table structure set and the target data table structure set to obtain difference information; splicing SQL statements based on the difference information to generate standard data manipulation statements; generating a structure modification script based on the standard data manipulation statements.

[0093] In this optional embodiment, the script file for modifying the target table structure generated from the statement change set can splice SQL statements to form a standard data manipulation statement (DML statement), and then write the formed DML statement into a new script file, which is the structure modification script for modifying the table structure information of the target data table structure set, so as to ensure that the table structure information of the target database is consistent with that of the source database.

[0094] In this way, it is possible to obtain the content differences between each table in the source database and the corresponding tables with the same table structure in the target database through the structural analysis of the migrated table structure, providing the possibility for accurately dividing the access traffic using these content differences in the subsequent process.

[0095] S13. Based on the statement change set and the clustering algorithm, divide the access traffic information of the source database to obtain the synchronization time set.

[0096] In an optional embodiment, dividing the access traffic information of the source database based on the statement change set and the clustering algorithm to obtain the synchronization time set includes:

[0097] S131. Use a traffic monitoring tool to obtain the access traffic information of the source database.

[0098] In this optional embodiment, the traffic monitoring tool of the source database can be enabled to obtain the access traffic information of the source database, and the log information and access traffic information of the database can be recorded in the access record table when the source database is running.

[0099] S132. Use the K-means clustering algorithm to divide the access traffic information, and the value of K is specified and obtained from the statement change set.

[0100] In this optional embodiment, the K-means clustering algorithm can be used to divide the access traffic information. Since the K-means clustering algorithm requires specifying the value of K in advance, and the value of K in this solution can be directly specified and obtained from the statement change set, the number of classifications obtained after dividing the access traffic information using the K-means clustering algorithm is consistent with the number of changes in the table structure information in the statement change set.

[0101] S133. Use the time period corresponding to the access traffic information of each category after division as the synchronization time period, and all the synchronization time periods constitute the synchronization time set.

[0102] In this optional embodiment, after dividing the access traffic information, use the time period corresponding to the access traffic information of each category after division as the synchronization time period, and all the synchronization time periods constitute the synchronization time set.

[0103] Exemplarily, there are 5 different field information in the table structures of the source database and the target database. Therefore, the value of K is set to 5, and the access traffic information is divided into 5 categories according to the traffic volume. Then, the time period corresponding to each category is used as the synchronization time period, and all the synchronization time periods corresponding to the 5 categories constitute the data synchronization time set.

[0104] In this way, the data to be migrated can be segmented and migrated during the time period that minimizes the impact on the normal operation of the system according to the access traffic of the source database, thereby improving the efficiency of the normal operation of the system.

[0105] S14, synchronize the data in the source database to the target database based on the statement change set and the synchronization time set to complete the data migration.

[0106] In this optional embodiment, the statement change set is sorted according to the amount of change information, and at the same time, the synchronization time set is sorted according to the traffic volume, so that the table structures correspond to the synchronization time periods of the categories in the order of decreasing change information in the statement change set and increasing traffic in the synchronization time set in sequence.

[0107] Exemplarily, there are 5 change information in total in the statement change set, namely A, B, C, D, and E. The change characters corresponding to each change information are 10, 15, 6, 8, and 20 respectively. After sorting according to the amount of change information, it is E, B, A, D, C; at the same time, the synchronization time periods obtained after dividing the access traffic are 0:00 - 8:00, 8:00 - 18:00, 18:00 - 20:00, 20:00 - 21:00, 21:00 to 24:00. After sorting according to the traffic volume in each time period, it is 8:00 - 18:00, 18:00 - 20:00, 20:00 - 21:00, 21:00 to 24:00, 0:00 - 8:00. Then, the time periods corresponding to E, B, A, D, and C are 0:00 - 8:00, 21:00 to 24:00, 20:00 - 21:00, 18:00 - 20:00, 8:00 - 18:00, that is, to ensure that the part with more change information and longer migration time in the database corresponds to the time period with less access traffic, so as to minimize the impact of data migration on the normal operation efficiency of the system.

[0108] In this optional embodiment, the user can set the mapping data of the data to be migrated in the source database to the target database in the web page. The program in the web can generate the corresponding migration file according to this mapping data and synchronize the data to the target database; when the data migration method is database migration, the program in the web can generate the corresponding migration file according to the information and its structure of the data to be migrated and migrate the data and its structure to the target database.

[0109] In this way, according to the amount of data to be migrated in each table of the source database, the corresponding migration time period can be allocated for it, thereby effectively reducing the impact of the database migration process on the normal operation efficiency of the system.

[0110] In this way, by obtaining the information of the source database to construct a table structure that conforms to the target database type, and matching the statement change information between the source database and the target database based on the table structure information as the data to be migrated, and at the same time combining the access traffic of the source database to segmentally migrate the data to be migrated within the time period that does not affect the normal operation of the system as much as possible, thereby effectively improving the efficiency of the normal operation of the system.

[0111] Please refer to Figure 4 , Figure 4 which is the functional module diagram of a preferred embodiment of the artificial intelligence-based data migration device of the present application. The artificial intelligence-based data migration device 11 includes an acquisition unit 110, a configuration unit 111, a matching unit 112, a partitioning unit 113, and a synchronization unit 114. The modules / units referred to in the present application refer to a series of computer-readable instruction segments that can be executed by a processor 13 and can complete fixed functions, and are stored in a memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0112] In an alternative embodiment, the acquisition unit 110 is configured to acquire the database information of the source database and the target database.

[0113] In an alternative embodiment, acquiring the database information of the source database and the target database includes:

[0114] Receiving the database information of the source database input by the user according to the web page, where the database information of the source database includes connection name, database type, schema, host IP, database name, port, user name, password, and other contents;

[0115] Receiving the database information of the target database input by the user according to the web page, where the database information of the target database includes database table name, synchronization method, connection name, database type, host IP, database name, port, user name, password, and target database table name, and other contents.

[0116] In an alternative embodiment, the source database is the source library for data synchronization, the source database can be multiple databases that conform to standard SQL, the target database is the target library for data synchronization, and the target database can be multiple databases that conform to standard SQL.

[0117] In this alternative embodiment, when a user needs to perform database migration, the user can open a web page and input configuration information on the web page to achieve the purpose of web-based database migration. First, the information of the source database that the user needs to input can be included on the opened web page. The information of the source database that needs to be input can specifically include: connection name, database type, schema, host IP, database name, port, username, password, and other contents.

[0118] In this alternative embodiment, after inputting the information of the source database, the user can determine some or all of the data tables in the source database and some data fields in the data tables as the data to be migrated according to the requirements. Among them, the methods for determining the data information to be migrated can include: selecting through the cursor, inputting the specific positions of the data tables or data fields to be migrated in the database to be migrated, and directly inputting the data content to be migrated.

[0119] In this alternative embodiment, after determining the information of the source database, the information of the target database can be input in another web page. In this web page, the information of the target database can include: database table name, synchronization method, connection name, database type, host IP, database name, port, username, password, and target database table name, etc. Among them, the information of the source database and the information of the target database are not exactly the same. The information of the target database depends on the specific type and configuration of the target database.

[0120] In an alternative embodiment, the configuration unit 111 is used to configure the table structures of the source database and the target database based on the database information to obtain the corresponding relationship of the table structures.

[0121] In an alternative embodiment, configuring the table structures of the source database and the target database based on the database information to obtain the corresponding relationship of the table structures includes:

[0122] Judging whether the types of the source database and the target database are the same based on the database information to obtain a judgment result, and the judgment result is the same and different;

[0123] If the judgment result is the same, then configure the corresponding relationship between the table structure of the source database and the table structure of the target database;

[0124] If the judgment result is different, then restructure the table structure of the source database into a format that conforms to the table structure of the target database, and then configure the corresponding relationship between the table structure of the source database and the table structure of the target database.

[0125] In an alternative embodiment, the database generally stores data in the form of a database table structure. The database table structure refers to the table structure of the database. The table structure consists of three parts: the table name, the fields in the table, and the records of the table. The table structure describes the framework of a database, defining the number of fields that make up a database, the name of each field, the data type of each field, and the length of each field, etc. Before creating the database, it is necessary to pre-design the data source table structure and store the relevant information of the table structure in the metadata of the database.

[0126] In this alternative embodiment, metadata is data about data, mainly information describing data properties, and supports functions such as indicating storage locations, historical data, resource lookup, file records, etc. In this embodiment, the metadata is saved in an Excel file and includes field information such as table names, group names, database names, information about the source table structure, and incremental timestamps.

[0127] In this alternative embodiment, based on the database information, the database environment information of the system operation can be obtained. Based on the database environment information, the database type can be obtained. The database environment information is saved in a configuration file and is obtained when creating a database connection.

[0128] In an alternative embodiment, the corresponding relationships of the table structure include information such as the corresponding relationships between data tables and the mapping relationships between data table fields. The database types of the source database and the target database can be different. For example, the source database is oracle and the target database is mysql.

[0129] In this alternative embodiment, if the judgment result is consistent, the table structures of the source database and the target database are associated through an association program, thereby completing the configuration of the corresponding relationship of the table structure. The association program is a preset script program used to find the corresponding table structure information in the target database based on information such as the table name and table fields in the table structure information of the source database and perform the association.

[0130] In this alternative embodiment, if the judgment result is inconsistent, a reorganization program is used to recombine the table structure of the source database according to the table structure of the target database to obtain a format that conforms to the table structure of the target database, and then the table structures of the source database and the target database are associated through an association program, thereby completing the configuration of the corresponding relationship of the table structure. The reorganization program is a preset script program used to recombine the table structure of the source database according to information such as the table name and table fields in the table structure information of the target database to obtain a format that conforms to the table structure of the target database.

[0131] A matching unit 112, configured to match the source database and the target database based on the corresponding relationship of the table structure to obtain a statement change set.

[0132] In an optional embodiment, matching the source database and the target database based on the corresponding relationship of the table structure to obtain a statement change set includes:

[0133] Performing a structure analysis on each data table in the source database and the target database to obtain a source data table structure set and a target data table structure set;

[0134] Based on the corresponding relationship of the table structure, comparing one by one whether the table structure information of the corresponding tables in the source data table structure set and the target data table structure set is consistent to obtain a statement change set.

[0135] In this optional embodiment, performing a structure analysis on each data table in the source database and the target database refers to obtaining the structure information of each data table, that is, obtaining the table name, the fields in the table, and the records of the table.

[0136] In this optional embodiment, by comparing whether there are differences in the field information of the source database table structure and the corresponding table structure of the target database, it is determined whether the source data table structure needs to change during data migration, that is, compared with the target database table structure, whether fields are added, whether fields are deleted, and whether the attributes of the fields (field name, field length, and data type of the field, etc.) are modified, and the finally matched different field information is used as the statement change set.

[0137] In an optional embodiment, matching the source database and the target database based on the data table structure set to obtain a statement change set includes: if the table structure information of the source data table structure set and the target data table structure set is consistent, the statement change set is an empty set; if the table structure information of the source data table structure set and the target data table structure set is inconsistent, a structure modification script is generated, and the table structure information of the target data table structure set is modified based on the structure modification script.

[0138] In an optional embodiment, if the table structure information of the source data table structure set and the target data table structure set is inconsistent, generating a structure modification script includes: analyzing the table structure information of the source data table structure set and the target data table structure set to obtain difference information; splicing sql statements based on the difference information to generate standard data manipulation statements; generating a structure modification script based on the standard data manipulation statements.

[0139] In this optional embodiment, the script file for modifying the target table structure generated by the statement change set can splice SQL statements to form a standard data manipulation statement (DML statement), and then write the formed DML statement into a new script file, which is the structure modification script for modifying the table structure information of the target data table structure set, so as to ensure that the table structure information of the target database is consistent with that of the source database.

[0140] In an optional embodiment, the partitioning unit 113 is configured to partition the access traffic information of the source database based on the statement change set and a clustering algorithm to obtain a synchronization time set.

[0141] In an optional embodiment, partitioning the access traffic information of the source database based on the statement change set and a clustering algorithm to obtain a synchronization time set includes:

[0142] Using a traffic monitoring tool to obtain the access traffic information of the source database;

[0143] Using the K-means clustering algorithm to partition the access traffic information, where the value of K is specified and obtained by the statement change set;

[0144] Taking the time period corresponding to each category of the partitioned access traffic information as the synchronization time period, and all the synchronization time periods constitute the synchronization time set.

[0145] In this optional embodiment, the K-means clustering algorithm can be used to partition the access traffic information. Since the K-means clustering algorithm requires specifying the value of K in advance, and the value of K in this solution can be directly specified and obtained by the statement change set, the number of classifications obtained after partitioning the access traffic information using the K-means clustering algorithm is consistent with the number of changes in the table structure information in the statement change set.

[0146] In this optional embodiment, after partitioning the access traffic information, taking the time period corresponding to each category of the partitioned access traffic information as the synchronization time period, and all the synchronization time periods constitute the synchronization time set.

[0147] Exemplarily, there are a total of 5 different field information in the table structures of the source database and the target database. Therefore, the value of K is set to 5, and the access traffic information is divided into 5 categories according to the traffic size. Then, the time period corresponding to each category is used as the synchronization time period, and all the synchronization time periods corresponding to the 5 categories constitute the data synchronization time set.

[0148] In an optional embodiment, the synchronization unit 114 is configured to synchronize the data in the source database to the target database based on the statement change set and the synchronization time set to complete data migration.

[0149] In this alternative embodiment, the set of statement changes is sorted according to the amount of change information, and at the same time, the set of synchronization times is sorted according to the traffic volume, so that the table structure corresponds to the synchronization time periods of the categories corresponding to the order of decreasing change information in the set of statement changes and the order of increasing traffic volume in the set of synchronization times in turn.

[0150] Exemplarily, there are a total of 5 change information items A, B, C, D, and E in the set of statement changes, and the corresponding changed characters for each change information item are 10, 15, 6, 8, and 20 in turn. After sorting according to the amount of change information, it is E, B, A, D, C; at the same time, the obtained synchronization time periods after dividing the access traffic are 0:00 - 8:00, 8:00 - 18:00, 18:00 - 20:00, 20:00 - 21:00, and 21:00 to 24:00. After sorting according to the traffic volume in each time period, it is 8:00 - 18:00, 18:00 - 20:00, 20:00 - 21:00, 21:00 to 24:00, 0:00 - 8:00. Then the time periods corresponding to E, B, A, D, and C are 0:00 - 8:00, 21:00 to 24:00, 20:00 - 21:00, 18:00 - 20:00, and 8:00 - 18:00, that is, to ensure that the part with more change information and longer migration time in the database corresponds to the time period with less access traffic, so as to minimize the impact of data migration on the normal operation efficiency of the system.

[0151] In this alternative embodiment, the user can set the mapping data of the data to be migrated in the source database to the target database in the web page. The program in the web can generate a corresponding migration file according to the mapping data and synchronize the data to the target database; when the data migration method is database migration, the program in the web can generate a corresponding migration file according to the information and its structure of the data to be migrated and migrate the data and its structure to the target database.

[0152] It can be seen from the above technical solutions that this application can construct a table structure that conforms to the type of the target database by obtaining the information of the source database, match the statement change information between the source database and the target database as the data to be migrated according to the table structure information, and at the same time, combine the access traffic of the source database to perform segmented migration of the data to be migrated during the time period that does not affect the normal operation of the system as much as possible, thereby effectively improving the efficiency of the normal operation of the system.

[0153] Please refer to Figure 5 , which is a schematic structural diagram of an electronic device provided by an embodiment of this application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement the data migration method based on artificial intelligence described in any of the above embodiments.

[0154] In an alternative embodiment, the electronic device 1 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as an artificial intelligence-based data migration program.

[0155] Figure 5 Only the electronic device 1 with the memory 12 and the processor 13 is shown. Those skilled in the art can understand that Figure 5 the shown structure does not constitute a limitation on the electronic device 1, and it may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0156] Combined with Figure 1 , the memory 12 in the electronic device 1 stores multiple computer-readable instructions to implement an artificial intelligence-based data migration method, and the processor 13 can execute the multiple instructions to implement:

[0157] Obtain the database information of the source database and the target database;

[0158] Configure the table structures of the source database and the target database based on the database information to obtain the corresponding relationship of the table structures;

[0159] Match the source database and the target database based on the corresponding relationship of the table structures to obtain a statement change set;

[0160] Divide the access traffic information of the source database based on the statement change set and a clustering algorithm to obtain a synchronization time set;

[0161] Synchronize the data in the source database to the target database based on the statement change set and the synchronization time set to complete the data migration.

[0162] Specifically, for the specific implementation method of the above instructions by the processor 13, reference can be made to Figure 1 the description of the relevant steps in the corresponding embodiment, which will not be elaborated here.

[0163] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 can be either a bus structure or a star structure. The electronic device 1 can also include more or fewer other hardware or software than shown, or different component arrangements. For example, the electronic device 1 can also include input / output devices, network access devices, etc.

[0164] It should be noted that the electronic device 1 is only an example. Other existing or future electronic products that can be adapted to this application should also be included in the protection scope of this application and are hereby incorporated by reference.

[0165] Among them, the memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as the mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed in the electronic device 1, such as the code of the data migration program based on artificial intelligence, etc., but also to temporarily store the data that has been output or will be output.

[0166] In some embodiments, the processor 13 can be composed of integrated circuits. For example, it can be composed of a single packaged integrated circuit, or can be composed of multiple integrated circuits with the same or different functions packaged, including a combination of one or more Central Processing Units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips, etc. The processor 13 is the control core (Control Unit) of the electronic device 1, connecting all components of the entire electronic device 1 through various interfaces and lines, and by running or executing programs or modules stored in the memory 12 (such as executing the data migration program based on artificial intelligence, etc.), and calling the data stored in the memory 12, to execute various functions of the electronic device 1 and process data.

[0167] The processor 13 executes the operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application programs to implement the steps in the above various embodiments of the data migration method based on artificial intelligence, such as Figures 1 to 3 the steps shown.

[0168] Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units can be a series of computer-readable instruction segments capable of completing specific functions, and these instruction segments are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program can be divided into an acquisition unit 110, a configuration unit 111, a matching unit 112, a division unit 113, and a synchronization unit 114.

[0169] The integrated unit implemented in the form of software function modules can be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium and include several instructions for causing a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute parts of the artificial intelligence-based data migration method described in various embodiments of the present application.

[0170] If the integrated module / unit of the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present application, it can also be completed by a computer program instructing relevant hardware devices. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented.

[0171] Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory, and other memories, etc.

[0172] Furthermore, the computer-readable storage medium mainly includes a storage program area and a storage data area. Among them, the storage program area can store an operating system, application programs required for at least one function, etc.; the storage data area can store data created according to the use of blockchain nodes, etc.

[0173] The blockchain referred to in the present application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, in essence, is a decentralized database, a string of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity (anti-counterfeiting) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0174] The bus can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, in Figure 5 only one arrow is used to represent it, but it does not mean that there is only one bus or one type of bus. The bus is arranged to implement connection communication between the memory 12 and at least one processor 13, etc.

[0175] Although not shown, the electronic device 1 may further include a power source (such as a battery) for powering each component. Preferably, the power source can be logically connected to the at least one processor 13 through a power management device, so as to implement functions such as charging management, discharging management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharge device, a power failure detection circuit, a power converter or inverter, and a power status indicator. The electronic device 1 may further include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0176] Furthermore, the electronic device 1 may further include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0177] Optionally, the electronic device 1 may further include a user interface. The user interface may be a display, an input unit (such as a keyboard), and optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display the information processed in the electronic device 1 and to display a visual user interface.

[0178] An embodiment of the present application further provides a computer-readable storage medium (not shown in the figure). Computer-readable instructions are stored in the computer-readable storage medium, and the computer-readable instructions are executed by a processor in the electronic device to implement the data migration method based on artificial intelligence described in any of the above embodiments.

[0179] It should be understood that the above embodiments are only for illustration purposes and are not limited by this structure in the scope of the application.

[0180] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation.

[0181] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, in each embodiment of the present application, the functional modules can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a combination of hardware and software functional modules.

[0183] Furthermore, it is obvious that the term "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described in the specification can also be implemented by one unit or device through software or hardware. The terms such as first and second are used to represent names and do not represent any specific order.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A data migration method based on artificial intelligence, characterized in that, Including: Obtain database information of the source database and the target database; Configure the table structures of the source database and the target database based on the database information to obtain the corresponding relationship of the table structures; Match the source database and the target database based on the corresponding relationship of the table structures to obtain a set of statement changes; Divide the access traffic information of the source database based on the set of statement changes and a clustering algorithm to obtain a set of synchronization times, including: using a traffic monitoring tool to obtain the access traffic information of the source database; using the K-means clustering algorithm to divide the access traffic information, where the value of K is specified by the set of statement changes; taking the time periods corresponding to each category of the divided access traffic information as synchronization time periods, and all the synchronization time periods constitute the set of synchronization times; the number of categories obtained after dividing the access traffic information is consistent with the number of changes in the table structure information in the set of statement changes; Synchronize the data in the source database to the target database based on the set of statement changes and the set of synchronization times to complete data migration, including: sorting the set of statement changes according to the amount of change information, and at the same time sorting the set of synchronization times according to the traffic size, and the table structures correspond to the synchronization time periods of the categories in ascending order of traffic in the set of synchronization times in the order of decreasing change information in the set of statement changes; among them, the part with more change information and longer migration time corresponds to the time period with less access traffic.

2. The data migration method based on artificial intelligence according to claim 1, wherein The obtaining of the database information of the source database and the target database includes: Receiving the database information of the source database input by the user according to the web page, and the database information of the source database includes connection name, database type, schema, host IP, database name, port, user name, and password; Receiving the database information of the target database input by the user according to the web page, and the database information of the target database includes database table name, synchronization method, connection name, database type, host IP, database name, port, user name, password, and target database table name.

3. The data migration method based on artificial intelligence according to claim 1, characterized in that, The configuring of the table structures of the source database and the target database based on the database information to obtain the corresponding relationship of the table structures includes: Judging whether the types of the source database and the target database are the same based on the database information to obtain a judgment result, and the judgment result is the same and different; If the judgment result is the same, then configure the corresponding relationship between the table structure of the source database and the table structure of the target database; If the judgment result is different, then reorganize the table structure of the source database into a format that conforms to the table structure of the target database, and then configure the corresponding relationship between the table structure of the source database and the table structure of the target database.

4. The data migration method based on artificial intelligence according to claim 1, wherein The matching of the source database and the target database based on the corresponding relationship of the table structures to obtain a set of statement changes includes: Conduct structure analysis on each data table in the source database and the target database to obtain a set of source data table structures and a set of target data table structures; Compare the table structure information of the corresponding tables in the source data table structure set and the target data table structure set one by one based on the above table structure correspondence relationship to obtain a statement change set.

5. The data migration method based on artificial intelligence according to claim 4, wherein The step of comparing the table structure information of the corresponding tables in the source data table structure set and the target data table structure set one by one based on the above table structure correspondence relationship to obtain a statement change set includes: If the table structure information of the source data table structure set and the target data table structure set is consistent, the statement change set is an empty set; If the table structure information of the source data table structure set and the target data table structure set is inconsistent, generate a structure modification script, and modify the table structure information of the target data table structure set based on the structure modification script.

6. The data migration method based on artificial intelligence according to claim 5, characterized in that The step of generating a structure modification script if the table structure information of the source data table structure set and the target data table structure set is inconsistent includes: Analyze the table structure information of the source data table structure set and the target data table structure set to obtain difference information; Concatenate SQL statements based on the difference information to generate standard data manipulation statements; Generate a structure modification script based on the standard data manipulation statements.

7. An artificial intelligence data migration device, the device comprising a unit for implementing the method according to any one of claims 1 to 6, characterized in that, including: An acquisition unit for acquiring database information of a source database and a target database; A configuration unit for configuring the table structures of the source database and the target database based on the database information to obtain a table structure correspondence relationship; A matching unit for matching the source database and the target database based on the table structure correspondence relationship to obtain a statement change set; A partitioning unit for partitioning the access traffic information of the source database based on the statement change set and a clustering algorithm to obtain a synchronization time set; A synchronization unit for synchronizing the data in the source database to the target database based on the statement change set and the synchronization time set to complete data migration.

8. An electronic device, characterized in that, including: A memory storing computer-readable instructions; and A processor that executes the computer-readable instructions stored in the memory to implement the artificial intelligence-based data migration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the artificial intelligence-based data migration method according to any one of claims 1 to 6 is implemented.

Images