Data migration method and apparatus, terminal device, and storage medium

Abstract

The application discloses a data migration method and device, terminal equipment and a storage medium. The data migration method is applied to a data migration system, the data migration system comprises a static data source and a plurality of dynamic data sources, and the data migration method comprises the following steps: based on migration configuration information of the static data source, performing data migration between the plurality of dynamic data sources. According to the application, the dynamic data source is used as a data migration-out party or a data migration-in party, and the static data source provides migration configuration information for controlling the data migration process. Under the data migration architecture, even if the migration configuration information is modified, the corresponding migration application does not need to be restarted, and the flexibility and efficiency of the data migration are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and particularly relates to a data migration method and device, a terminal device and a storage medium. BACKGROUND

[0002] Database cloud refers to deploying a database in the cloud so that users can access and use the database through the Internet. This method can improve the availability, scalability and security of the database, while reducing maintenance costs. In order to realize database cloud, the data of different databases needs to be migrated, for example, the data of a SQLServer database is migrated to a MySQL database.

[0003] In the migration process, the database can also be regarded as a data source. At present, a static data source is usually taken as a migration-in party, which is a data source recording migration configuration information. In this migration mode, the static data source is responsible for receiving migrated data and providing migration configuration information to control the migration process. If it is necessary to modify the migration configuration information to adapt to different migration requirements, the corresponding migration application needs to be restarted, which leads to low migration efficiency. SUMMARY

[0004] The main purpose of the present application is to provide a data migration method, device, terminal device and storage medium, aiming at solving or improving the problem of low data migration efficiency at present.

[0005] To achieve the above purpose, the present application provides a data migration method, which is applied to a data migration system, the data migration system comprising a static data source and a plurality of dynamic data sources, and the data migration method comprising:

[0006] Based on the migration configuration information of the static data source, data migration is performed between the plurality of dynamic data sources.

[0007] Optionally, before the step of controlling the data migration between the plurality of dynamic data sources based on the migration configuration information of the static data source, the method further comprises:

[0008] Based on a preset resource identifier, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in are concatenated respectively to obtain a corresponding migration configuration parameter group;

[0009] The migration configuration parameter group is stored in the static data source.

[0010] Optionally, before the step of controlling the data migration between the plurality of dynamic data sources based on the migration configuration information of the static data source, the method further comprises:

[0011] In response to a first type of selection operation of a user through a preset visual migration task setting interface, determine the dynamic data source to be migrated out and the dynamic data source to be migrated in;

[0012] Query the dynamic data source to be migrated out and the dynamic data source to be migrated in, respectively to obtain the selected migration out table information and the selected migration in table information;

[0013] Display the selected migration out table information and the selected migration in table information on the preset visual migration task setting interface;

[0014] In response to a second type of selection operation of a user through the visual migration task setting interface, determine the target migration out table of the dynamic data source to be migrated out and the target migration in table of the dynamic data source to be migrated in;

[0015] Create a corresponding migration task based on the target migration out table and the target migration in table;

[0016] Store the migration task in the static data source.

[0017] Optionally, the step of migrating data between the plurality of dynamic data sources based on the migration configuration information of the static data source comprises:

[0018] Migrate the data of the target migration out table to the target migration in table based on the migration configuration parameters and the migration task of the static data source.

[0019] Optionally, the step of migrating the data of the target migration out table to the target migration in table based on the migration configuration parameters and the migration task of the static data source comprises:

[0020] Query the target data of the target migration out table based on the migration configuration parameters and the migration task of the static data source;

[0021] Insert and update the target data into the target migration in table based on a preset insert update statement.

[0022] Optionally, after the step of migrating data between the plurality of dynamic data sources based on the migration configuration information of the static data source, the method further comprises:

[0023] Based on a preset total number of data table checking rule and a table data spot check rule, perform self-checking on the migration out table and the migration in table to obtain a corresponding self-checking result.

[0024] Optionally, the migration configuration information comprises a migration task, and the step of migrating data between the plurality of dynamic data sources based on the migration configuration information of the static data source comprises:

[0025] In the process of performing the migration task, it is determined whether a migration configuration parameter group corresponding to the resource identifier exists based on a preset aspect judgment rule and the resource identifier corresponding to the migration task;

[0026] If not, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in respectively specified by the migration task are concatenated based on a preset pointcut annotation to obtain a corresponding migration configuration parameter group, and the migration configuration parameter group is stored in the static data source.

[0027] The resource identifier is stored in a local thread variable of the thread corresponding to the migration task;

[0028] The corresponding migration configuration parameter group is obtained based on the resource identifier in the local thread variable by the thread, and data migration is performed between the dynamic data source to be migrated out and the dynamic data source to be migrated in based on the migration configuration parameters and the migration task.

[0029] Embodiments of the present application also propose a data migration device, which is applied to a data migration system, the data migration system comprising a static data source and a plurality of dynamic data sources, and the data migration device comprising:

[0030] A migration module is configured to perform data migration between the plurality of dynamic data sources based on migration configuration information of the static data source.

[0031] Embodiments of the present application also propose a terminal device, which comprises a memory, a processor, and a data migration program stored in the memory and executable on the processor, and the data migration program, when executed by the processor, implements the steps of the data migration method described above.

[0032] Embodiments of the present application also propose a computer-readable storage medium, which stores a data migration program, and the data migration program, when executed by a processor, implements the steps of the data migration method described above.

[0033] The data migration method, device, terminal device, and storage medium proposed by embodiments of the present application are applied to a data migration system, the data migration system comprising a static data source and a plurality of dynamic data sources, and the data migration method comprising performing data migration between the plurality of dynamic data sources based on migration configuration information of the static data source. Based on the present application, a dynamic data source is used as a data migration out party or a data migration in party, and a static data source provides migration configuration information for controlling the data migration process. Under this data migration architecture, even if the migration configuration information is modified, the corresponding migration application does not need to be restarted, effectively improving the flexibility and efficiency of data migration. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 Figure 1 is a schematic diagram of a functional module of a terminal device to which the data migration device of the present application belongs;

[0035] Figure 2 Figure 2 is a schematic diagram of a flow of a first exemplary embodiment of the data migration method of the present application;

[0036] Figure 3 Figure 3 is a schematic diagram of a functional architecture of a data migration system involved in the data migration method of the present application;

[0037] Figure 4 Figure 4 is a schematic diagram of a data source of the data migration system involved in the data migration method of the present application;

[0038] Figure 5 Figure 5 is a schematic diagram of a flow of a second exemplary embodiment of the data migration method of the present application;

[0039] Figure 6 Figure 6 is a schematic diagram of a flow of a third exemplary embodiment of the data migration method of the present application;

[0040] Figure 7 Figure 7 is a schematic diagram of a flow of creation of a migration task involved in the data migration method of the present application;

[0041] Figure 8 Figure 8 is a schematic diagram of a flow of a fourth exemplary embodiment of the data migration method of the present application;

[0042] Figure 9 Figure 9 is a schematic diagram of execution of a migration task involved in the data migration method of the present application;

[0043] Figure 10 Figure 10 is a schematic diagram of a flow of a fifth exemplary embodiment of the data migration method of the present application;

[0044] Figure 11 Figure 11 is a schematic diagram of a flow of a sixth exemplary embodiment of the data migration method of the present application;

[0045] Figure 12 Figure 12 is a schematic diagram of a flow of a seventh exemplary embodiment of the data migration method of the present application;

[0046] Figure 13 Figure 13 is a schematic diagram of a dynamic data source registration scheme involved in the data migration method of the present application.

[0047] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein merely exemplify the present application and are not intended to limit the present application.

[0049] The main solution of the embodiment of the application is that the data migration method is applied to a data migration system, the data migration system comprises a static data source and a plurality of dynamic data sources, and the data migration method comprises: performing data migration among the plurality of dynamic data sources based on migration configuration information of the static data source. Based on the solution of the application, the dynamic data source is used as a data migration out party or a data migration in party, and the static data source provides migration configuration information for controlling the data migration process. Under this data migration architecture, even if the migration configuration information is modified, the corresponding migration application does not need to be restarted, and the flexibility and efficiency of data migration are effectively improved.

[0050] Specifically, referring to Figure 1 , Figure 1 is a schematic diagram of a functional module of a terminal device to which the data migration device of the application belongs. The data migration device can be a device independent of the terminal device and capable of data migration, which can be carried on the terminal device in the form of hardware or software. The terminal device can be a smart mobile terminal such as a mobile phone or a tablet computer having a data processing function, and can also be a fixed terminal device or a server having a data processing function.

[0051] In the embodiment, the terminal device to which the data migration device belongs comprises at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0052] The memory 130 stores an operating system and a data migration program, the data migration device can store migration configuration information based on a static data source, data migrated among a plurality of dynamic data sources, and the like in the memory 130, and the output module 110 can be a display screen or the like. The communication module 140 can include a WIFI module, a mobile communication module, and a Bluetooth module, and the like, and communicates with external devices or servers through the communication module 140.

[0053] When the data migration program in the memory 130 is executed by the processor, the following steps are implemented:

[0054] Perform data migration among the plurality of dynamic data sources based on the migration configuration information of the static data source.

[0055] Further, when the data migration program in the memory 130 is executed by the processor, the following steps are implemented:

[0056] Based on a preset resource identifier, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in are concatenated respectively to obtain a corresponding migration configuration parameter group;

[0057] The migration configuration parameter group is stored in the static data source.

[0058] Further, the data migration program in the memory 130, when executed by the processor, further implements the following steps:

[0059] In response to a first type of selection operation of the user through the preset visual migration task setting interface, determining the dynamic data source to be migrated out and the dynamic data source to be migrated in;

[0060] Querying the dynamic data source to be migrated out and the dynamic data source to be migrated in, respectively obtaining the selected migration out table information and the selected migration in table information;

[0061] Displaying the selected migration out table information and the selected migration in table information on the preset visual migration task setting interface;

[0062] In response to a second type of selection operation of the user through the visual migration task setting interface, determining the target migration out table of the dynamic data source to be migrated out and the target migration in table of the dynamic data source to be migrated in;

[0063] Creating a corresponding migration task based on the target migration out table and the target migration in table;

[0064] Storing the migration task in the static data source.

[0065] Further, the data migration program in the memory 130, when executed by the processor, further implements the following steps:

[0066] Based on the migration configuration parameters and the migration task of the static data source, migrating the data of the target migration out table to the target migration in table.

[0067] Further, the data migration program in the memory 130, when executed by the processor, further implements the following steps:

[0068] Based on the migration configuration parameters and the migration task of the static data source, querying to obtain the target data of the target migration out table;

[0069] Based on the preset insert update statement, inserting and updating the target data to the target migration in table.

[0070] Further, the data migration program in the memory 130, when executed by the processor, further implements the following steps:

[0071] Based on the preset total number of data table checking rule and the table data spot check rule, performing self-checking on the migration out table and the migration in table to obtain a corresponding self-checking result.

[0072] Further, the data migration program in the memory 130, when executed by the processor, further implements the following steps:

[0073] In the process of executing the migration task, whether the migration configuration parameter group corresponding to the resource identifier exists is determined based on a preset aspect judgment rule and the resource identifier corresponding to the migration task;

[0074] If not, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in respectively specified by the migration task are concatenated based on a preset pointcut annotation to obtain a corresponding migration configuration parameter group, and the migration configuration parameter group is stored in the static data source;

[0075] The resource identifier is stored in a local thread variable of the thread corresponding to the migration task;

[0076] The corresponding migration configuration parameter group is obtained by the thread based on the resource identifier in the local thread variable, and data migration is performed between the dynamic data source to be migrated out and the dynamic data source to be migrated in based on the migration configuration parameter and the migration task.

[0077] In the embodiment, the dynamic data source is used as the data migration out party or the data migration in party, and the static data source provides migration configuration information for controlling the data migration process. Under this data migration architecture, even if the migration configuration information is modified, the corresponding migration application does not need to be restarted, effectively improving the flexibility and efficiency of data migration.

[0078] Referring to Figure 2 , a flowchart is provided in the first embodiment of the data migration method. The data migration method is applied to a data migration system, the data migration system includes a static data source and a plurality of dynamic data sources, and the data migration method includes:

[0079] Step S10, based on the migration configuration information of the static data source, data migration is performed between the plurality of dynamic data sources.

[0080] Specifically, in order to realize database cloud migration, the data of different databases needs to be migrated, for example, the data of a SQLServer database is migrated to a MySQL database. In the migration process, the database can also be regarded as a data source. At present, a static data source is usually used as a migration in party, and the static data source is a data source recording migration configuration information. In this migration mode, the static data source is responsible for receiving migrated data and providing migration configuration information to control the migration process. If the migration configuration information needs to be modified to adapt to different migration requirements, the corresponding migration application needs to be restarted, which results in low migration efficiency.

[0081] To this end, the embodiment proposes a data migration system, which comprises one static data source and a plurality of dynamic data sources. The static data source corresponds to a relational database and records migration configuration information for controlling the migration process. The dynamic data sources can be divided into dynamic data sources to be migrated out and dynamic data sources to be migrated in. Figure 3 Figure 3 As shown in the figure,

[0082] As shown in the figure, Figure 4 Figure 4 The static data source comprises a dynamic data source configuration table, a migration task table and a migration log table. Each record in the dynamic data source configuration table corresponds to a resource ID and represents a migration pair, which is composed of a dynamic data source to be migrated out and a dynamic data source to be migrated in. Each record comprises the connection information, username and password of the dynamic data source to be migrated out (i.e. the source database) and the connection information, username and password of the dynamic data source to be migrated in (i.e. the target database). It can be understood that each record in the dynamic data source configuration table comprises a set of migration configuration parameters of the corresponding migration pair. In addition, each record in the migration task table corresponds to a migration task, and the granularity of the migration task is a table, i.e. the migration task performs data migration in the table dimension. Furthermore, the migration log table is used to record the execution result, the number of affected rows, the execution time consumption and the self-checking result of the migration task.

[0083] The dynamic data sources can be divided into dynamic data sources to be migrated out and dynamic data sources to be migrated in. As shown in the figure, Figure 3 The dynamic data source to be migrated out is dynamic data source 1, which corresponds to the database migration source database, and its database type is SQLServer. The dynamic data source to be migrated in is dynamic data source 2, which corresponds to the database migration target database, and its database type is MySQL. When creating a migration task, the source database information and the target database information corresponding to the resource ID are instantiated as data sources (DataSource) and managed using a connection pool. In addition, if the migration task spans N database migration source databases, there are N SQLServer database connection pools in the application at the same time. If the migration task spans N database migration target databases, there are N MySQL database connection pools in the application at the same time. ​​

[0084] The above examples of the database types of SQL Server / MySQL are only for better understanding of the present scheme, and the types of the databases corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in are not limited to SQL Server and MySQL.

[0085] The migration configuration information of the static data source includes the migration parameter group and the migration task corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in, and based on the migration parameter group and the migration task corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in, the data of the dynamic data source to be migrated out can be migrated to the dynamic data source to be migrated in.

[0086] In the embodiment, the dynamic data source is used as the migration out party or the migration in party of data, and the static data source provides the migration configuration information for controlling the data migration process. Under the data migration architecture, even if the migration configuration information is modified, the corresponding migration application does not need to be restarted, and the flexibility and efficiency of data migration are effectively improved.

[0087] Further, with reference to Figure 5 , the second embodiment of the data migration method of the present application provides a flowchart, based on the above Figure 2 , step S10, based on the migration configuration information of the static data source, before controlling the data migration between the plurality of dynamic data sources, further includes:

[0088] Step S01, based on the preset resource identifier, concatenating the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in respectively to obtain the corresponding migration configuration parameter group.

[0089] Step S02, storing the migration configuration parameter group in the static data source.

[0090] Specifically, based on the preset resource identifier, concatenating the connection information, username and password of the dynamic data source to be migrated out (i.e. the source database) and the connection information, username and password of the dynamic data source to be migrated in (i.e. the target database) to obtain the corresponding migration configuration parameter group, and recording the migration configuration parameter group and the resource identifier in the dynamic data source configuration table of the static data source. That is, the migration configuration parameter includes the connection information, username and password of the corresponding dynamic data source (database), and the migration configuration parameter group includes the connection information, username and password of the corresponding migration pair.

[0091] Each record in the dynamic data source configuration table corresponds to a resource ID and represents a migration pair, and a migration pair is composed of a dynamic data source to be migrated out and a dynamic data source to be migrated in. A record includes the connection information, username, and password of the dynamic data source to be migrated out (i.e., the source database), and the connection information, username, and password of the dynamic data source to be migrated in (i.e., the target database). It can be understood that each record in the dynamic data source configuration table includes a migration configuration parameter group of the corresponding migration pair.

[0092] In the embodiment, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in are concatenated based on the preset resource ID, to obtain a corresponding migration configuration parameter group. The migration configuration parameter group is stored in the static data source. In the embodiment, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in are concatenated based on the resource ID, to ensure the data consistency and integrity of the data migration process. In addition, the resource ID can confirm the migration pair relationship between the dynamic data source to be migrated out and the dynamic data source to be migrated in, to ensure that each step of the data migration can be effectively performed.

[0093] Further, with reference to Figure 6 , the third embodiment of the data migration method of the present application provides a flowchart. Based on the above Figure 5 , the step S10, based on the migration configuration information of the static data source, before performing the data migration between the dynamic data sources, further includes:

[0094] The step S03, in response to the first type of selection operation of the user through the preset visual migration task setting interface, determines the dynamic data source to be migrated out and the dynamic data source to be migrated in.

[0095] Specifically, as shown in Figure 7 , the step S03, in response to the first type of selection operation of the user through the preset visual migration task setting interface, determines the dynamic data source to be migrated out and the dynamic data source to be migrated in. Figure 7A flowchart of a migration task created by a data migration method of the present application is provided. The embodiment provides a visual migration task setting interface based on an online configuration interface of a data migration system. A user can determine a dynamic data source to be migrated out and a dynamic data source to be migrated in through a first type of selection operation on the preset visual migration task setting interface. For example, the visual migration task setting interface provides a dynamic data source to be migrated out drop-down selection menu and a dynamic data source to be migrated in drop-down selection menu, and the user can determine the dynamic data source to be migrated out and the dynamic data source to be migrated in from a plurality of dynamic data sources of the data migration system through drop-down selection. After determining the dynamic data source to be migrated out and the dynamic data source to be migrated in, the data migration system first checks whether the corresponding data source has been configured. If the corresponding data source has not been configured, the data source needs to be configured first before returning to the interface to create a migration task. If the corresponding data source has been configured, the user is allowed to continue creating a migration task.

[0096] Step S04: querying the dynamic data source to be migrated out and the dynamic data source to be migrated in to obtain selected migration-out table information and selected migration-in table information, respectively.

[0097] Specifically, the migration task is based on table dimension, and a single migration task supports migration of data of a single source table of the dynamic data source to be migrated out to a single target table of the dynamic data source to be migrated in, and migration of each table is a migration task. Therefore, the data migration system queries the dynamic data source to be migrated out and the dynamic data source to be migrated in to obtain selected migration-out table information and selected migration-in table information, respectively.

[0098] Step S05: displaying the selected migration-out table information and the selected migration-in table information on the preset visual migration task setting interface.

[0099] Specifically, in order to facilitate user selection of migration-out tables and migration-in tables, the selected migration-out table information and the selected migration-in table information can be displayed on the preset visual migration task setting interface. The selected migration-out table information and the selected migration-in table information are preferably displayed in the form of a drop-down menu, which can be a search-enabled drop-down menu. Similarly, the selected migration-out table information and the selected migration-in table information can also be displayed in the form of a single selection box, a multiple selection box, a single selection button, a multiple selection button, a slider, a switch, etc. for user selection operation.

[0100] Step S06: determining a target migration-out table of the dynamic data source to be migrated out and a target migration-in table of the dynamic data source to be migrated in in response to a second type of selection operation of the user on the visual migration task setting interface.

[0101] Specifically, the visual migration task settings interface displays multiple selectable tables for both the outgoing and incoming data sources. Users can perform a second type of selection operation through this interface, choosing the target outgoing and incoming tables from these multiple selectable tables. Correspondingly, the data migration system responds to the user's selection by determining the target outgoing table for the dynamic data source to be migrated out and the target incoming table for the dynamic data source to be migrated into.

[0102] Step S07: Create corresponding migration tasks based on the target migration-out table and the target migration-in table.

[0103] Specifically, after identifying the dynamic data source to be migrated out, the target migration out table, the dynamic data source to be migrated into, and the target migration in table, the user can proceed with a confirmation and submission operation. After the user's confirmation and submission, the data migration system will perform further verification. If the verification fails, the system will prompt the user to reselect the target migration in table; if the verification succeeds, the system will create a corresponding migration task based on the target migration out table and the target migration in table.

[0104] It is worth noting that, such as Figure 3 As shown, after selecting the target table from the drop-down menu corresponding to the candidate table to be migrated out in the user's visual migration task settings interface, migration options can be configured as needed. These options include field mapping, source table ignored fields, migration data range, and limits on the number of word queries / word insertions. Configuring the source table ignored fields and the migration data range can reduce data redundancy and resource consumption after migration.

[0105] Step S08: Store the migration task in the static data source.

[0106] Specifically, after creating a migration task, the migration task can be stored in the migration task table of the static data source. One record in the migration task table represents one migration task.

[0107] The embodiment provides a visual migration task setting interface, and a user can create a table dimension migration task through the visual migration task setting interface, so that the data migration system automatically executes the migration task, completes the data migration work, and improves the data migration efficiency. In addition, the user can also create migration tasks with different parameters according to actual needs, to control the specific process of data migration, and improve the controllability of the data migration process.

[0108] Further, with reference to Figure 8 , the fourth embodiment of the data migration method of the present application provides a flowchart, based on the above Figure 6 , step S10, based on the migration configuration information of the static data source, further refining the data migration between the plurality of dynamic data sources, including:

[0109] Step S101, based on the migration configuration parameters and the migration task of the static data source, migrating the data of the target migration-out table to the target migration-in table.

[0110] Specifically, as shown in Figure 9 , Figure 9 is a migration task execution schematic diagram involved in the data migration method of the present application. In the process of executing the migration task, first, a task identification (task ID) is generated, a thread resource is obtained from a thread pool, and then corresponding task parameters (TaskParam) are obtained. To ensure the validity of the task parameters, the task parameters are checked at this time, if the check fails, the execution of the migration task is ended; if the check passes, further data migration is performed based on the migration configuration parameters and the double-loop mode. In the double-loop mode, the outer loop is to query target data from the target migration-out table in sections, and the inner loop is responsible for inserting and updating the target data into the target migration-in table in sections. The section is based on the database single query quantity (dbQrySize) and the single insertion quantity (dbInsSize) in the task parameters. It is worth noting that by adjusting the single query quantity and the single insertion quantity to appropriate values, the migration speed can be improved.

[0111] After the migration task is executed, the corresponding task state can be updated.

[0112] The embodiment introduces the above scheme, and specifically introduces the migration configuration parameter and the migration task based on the static data source to migrate the data of the target migration-out table to the target migration-in table. In the embodiment, the migration configuration parameter and the migration task based on the static data source can migrate the data of the target migration-out table to the target migration-in table, realize automatic data migration in the table dimension, and improve the data migration efficiency.

[0113] Further, referring to Figure 10 , the fifth embodiment of the data migration method of the application provides a flowchart, based on the above Figure 8 , the step S101 introduces the migration configuration parameter and the migration task based on the static data source to migrate the data of the target migration-out table to the target migration-in table, which is further refined and includes:

[0114] The step S1011 introduces the migration configuration parameter and the migration task based on the static data source to query the target data of the target migration-out table.

[0115] The step S1012 introduces the target data into the target migration-in table based on the preset insert update statement.

[0116] Specifically, in the double-loop mode, the outer loop is to query the target data from the target migration-out table in sections, and the inner loop is responsible for inserting and updating the target data into the target migration-in table in sections. For the dynamic data source to be migrated into the MySQL database type, the replace into insert update statement can be used to execute the above insert update action. The replace into insert update statement has the following characteristics: if the data to be inserted already exists in the table (judged according to the primary key or the unique index), the original data is deleted first, and then the new data is inserted; otherwise, the new data is directly inserted. The replace into insert update statement can avoid data duplication or conflict.

[0117] The embodiment introduces the above scheme, and specifically introduces the migration configuration parameter and the migration task based on the static data source to query the target data of the target migration-out table; and introduces the target data into the target migration-in table based on the preset insert update statement. In the embodiment, the insert update statement is introduced into the data migration process, which can effectively avoid the occurrence of data duplication or conflict, and ensure the consistency of the migrated data.

[0118] Further, referring to Figure 11 , the sixth embodiment of the data migration method of the application provides a flowchart, based on the above Figure 8In the embodiment shown, step S10, after data migration between the plurality of dynamic data sources based on the migration configuration information of the static data source, further includes:

[0119] Step S09, based on the preset data table total number checking rule and the table data sampling rule, self-inspects the migration-out table and the migration-in table to obtain a corresponding self-inspection result.

[0120] Specifically, as shown in Figure 9 After the migration task is executed and the task state is updated, task self-inspection can be performed, and the process of task self-inspection is based on the preset data table total number checking rule and the table data sampling rule.

[0121] Under the data table total number checking rule, the data table total number corresponding to the target migration-out table and the target migration-in table is queried respectively, and whether the data table total number corresponding to the target migration-out table and the target migration-in table is equal is compared to obtain a data table total number checking result.

[0122] Under the table data sampling rule, N (configurable) pieces of sampling data of the target migration-out table are randomly sampled, and N pieces of sampling data of the target migration-in table are queried according to the primary keys of the N pieces of sampling data of the target migration-out table. Further, according to the field mapping method configured by the task, whether the field values of the sampling data of the target migration-out table and the sampling data of the target migration-in table are the same is compared to obtain a table data sampling result.

[0123] On this basis, according to the data table total number checking result and the table data sampling result, a corresponding self-inspection result can be comprehensively obtained, and the self-inspection result is helpful for the user to determine whether the migration task execution result meets the expectation and to identify potential risk points. At the same time, the self-inspection result is also stored in the corresponding migration log table.

[0124] The embodiment obtains the self-inspection result by performing self-inspection on the migration-out table and the migration-in table based on the preset data table total number checking rule and the table data sampling rule. In the embodiment, the self-inspection result is obtained after the migration task is executed, the user can determine whether the self-inspection result meets the expectation, and can also identify potential risk points according to the self-inspection result, thereby improving the controllability of the data migration process.

[0125] Further, referring to Figure 12 , the seventh embodiment of the data migration method of the present application provides a flowchart, which is based on the above Figure 8 In the embodiment shown, the migration configuration information includes a migration task, and step S10, based on the migration configuration information of the static data source, further refines the data migration between the plurality of dynamic data sources, including:

[0126] Step S102, in the process of executing the migration task, based on the preset aspect judgment rule and the resource identifier corresponding to the migration task, it is determined whether the migration configuration parameter group corresponding to the resource identifier exists.

[0127] Step S103, if not, based on the preset pointcut annotation, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in specified by the migration task are concatenated respectively to obtain the corresponding migration configuration parameter group, and the migration configuration parameter group is stored in the static data source.

[0128] Step S104, the resource identifier is stored in the local thread variable of the thread corresponding to the migration task.

[0129] Step S105, through the thread, the corresponding migration configuration parameter group is obtained based on the resource identifier in the local thread variable, and based on the migration configuration parameter and the migration task, data migration is performed between the dynamic data source to be migrated out and the dynamic data source to be migrated in.

[0130] Specifically, the most important prerequisite for correct execution of the migration task is to correctly obtain the connection information of the dynamic data source to be migrated out and the connection information of the dynamic data source to be migrated in. Since the to-be-migrated-out party and the to-be-migrated-in party of the embodiment are both dynamic data sources, how to identify the dynamic data source corresponding to the current migration task in multiple migration tasks and obtain the corresponding connection information is the key and difficulty.

[0131] Therefore, the embodiment proposes a dynamic data source registration scheme based on AOP (Aspect Oriented Programming, Aspect Oriented Programming) technology, as shown in Figure 13 Figure 13 is a schematic diagram of the dynamic data source registration scheme involved in the data migration method of the present application. The data source (DataSource) and the thread are bound, the configuration of the dynamic data source can be implemented by inheriting the AbstractRoutingDataSource class, the key value of the data source of the current thread is obtained through the AbstractRoutingDataSource class, and the key value of the data source is saved through the local thread variable (ThreadLocal) of the current thread. The key value of the data source is changed to determine which database connection is generated, and the connection is returned to the application from the connection pool.

[0132] ​According to the principle, the data migration system adopts the AOP technology, and customizes the pointcut annotations @MigrateSource and @MigrateTarget. All the methods related to the business logic layer (Service layer) of the database interaction corresponding to the dynamic data source to be migrated are added with the pointcut annotation @MigrateSource of the dynamic data source to be migrated, all the methods related to the business logic layer (Service layer) of the database interaction corresponding to the dynamic data source to be migrated are added with the pointcut annotation @MigrateTarget of the dynamic data source to be migrated, and the resource identifier is transmitted in the first parameter of the method, that is, the migration configuration parameters corresponding to the dynamic data source to be migrated and the dynamic data source to be migrated are concatenated based on the preset pointcut annotations.

[0133] Under the preset aspect judgment rule, in the beforeAspect method, it is judged whether the migration configuration parameter group corresponding to the resource identifier corresponding to the current migration task exists, and if not, a database connection pool (HikariDataSource) is created, the migration configuration parameter group is dynamically woven, and the resource identifier corresponding to the current migration task is stored in the local thread variable of the corresponding thread. In this way, the thread can obtain the corresponding migration configuration parameter group for data migration. At this point, the target of dynamically switching the dynamic data source to be migrated and the dynamic data source to be migrated at the same time has been achieved.

[0134] The embodiment achieves the above-mentioned scheme, specifically by determining whether the migration configuration parameter group corresponding to the resource identifier exists based on the preset aspect judgment rule and the resource identifier corresponding to the migration task during the execution of the migration task; if not, concatenating the migration configuration parameters corresponding to the dynamic data source to be migrated and the dynamic data source to be migrated respectively specified by the migration task based on the preset pointcut annotations, obtaining the corresponding migration configuration parameter group, and storing the migration configuration parameter group in the static data source; storing the resource identifier in the local thread variable of the thread corresponding to the migration task; obtaining the corresponding migration configuration parameter group based on the resource identifier in the local thread variable through the thread, and performing data migration between the dynamic data source to be migrated and the dynamic data source to be migrated based on the migration configuration parameters and the migration task. In the embodiment, the AOP technology is used to dynamically switch the dynamic data source to be migrated and the dynamic data source to be migrated, which can meet the diversified data migration requirements and improve the controllability of the data migration process.

[0135] Furthermore, the data migration device is applied to a data migration system, the data migration system comprises a static data source and a plurality of dynamic data sources, and the data migration device comprises:

[0136] a migration module, configured to perform data migration among the plurality of dynamic data sources based on migration configuration information of the static data source.

[0137] The principle and implementation process of the data migration are achieved, and details are not described herein again.

[0138] Furthermore, the data migration device is applied to a data migration system, the data migration system comprises a static data source and a plurality of dynamic data sources, and the data migration device comprises:

[0139] Since the data migration program is executed by the processor, all the technical solutions of all the foregoing embodiments are adopted, and at least all the beneficial effects brought by all the technical solutions of all the foregoing embodiments are achieved, and details are not described herein again.

[0140] Furthermore, the data migration device is applied to a data migration system, the data migration system comprises a static data source and a plurality of dynamic data sources, and the data migration device comprises:

[0141] Since the data migration program is executed by the processor, all the technical solutions of all the foregoing embodiments are adopted, and at least all the beneficial effects brought by all the technical solutions of all the foregoing embodiments are achieved, and details are not described herein again.

[0142] Compared with the prior art, the data migration method, device, terminal device and storage medium provided by the embodiments of the present application perform data migration among the plurality of dynamic data sources based on migration configuration information of the static data source. Based on the present application, the dynamic data source is used as a data migration out party or a data migration in party, and the static data source provides migration configuration information for controlling the data migration process. Under the data migration architecture, even if the migration configuration information is modified, the corresponding migration application does not need to be restarted, and the flexibility and efficiency of data migration are effectively improved.

[0143] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0144] The above-mentioned sequence numbers of embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0145] Those skilled in the art can clearly understand the above-mentioned embodiment method from the description of the above embodiments, which can be realized by software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, controlled terminal, or network device) execute the method of each embodiment of the present application.

[0146] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data migration method, characterized by, The data migration method is applied to a data migration system, the data migration system comprising a static data source and a plurality of dynamic data sources, and the data migration method comprising: based on the migration configuration information of the static data source, performing data migration among the plurality of dynamic data sources, wherein when the migration configuration information is modified, the migration application corresponding to the migration configuration information does not need to be restarted; wherein, based on the migration configuration information of the static data source, before the step of controlling the data migration among the plurality of dynamic data sources, it further comprises: based on the preset resource identifier, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in are concatenated to obtain the corresponding migration configuration parameter group; storing the migration configuration parameter group in the static data source; wherein, the migration configuration information comprises a migration task, and the step of performing data migration among the plurality of dynamic data sources based on the migration configuration information of the static data source comprises: in the process of executing the migration task, based on the preset aspect judgment rule and the resource identifier corresponding to the migration task, it is determined whether the migration configuration parameter group corresponding to the resource identifier exists; if not, based on the preset pointcut annotation, the migration configuration parameters corresponding to the dynamic data source to be migrated out and the dynamic data source to be migrated in specified by the migration task are concatenated to obtain the corresponding migration configuration parameter group, and the migration configuration parameter group is stored in the static data source; storing the resource identifier in the local thread variable of the thread corresponding to the migration task; based on the resource identifier in the local thread variable, the corresponding migration configuration parameter group is obtained by the thread, and based on the migration configuration parameter and the migration task, data migration is performed between the dynamic data source to be migrated out and the dynamic data source to be migrated in.

2. The data migration method of claim 1, wherein, Before the step of performing data migration among the plurality of dynamic data sources based on the migration configuration information of the static data source, it further comprises: in response to the user's first type of selection operation through the preset visual migration task setting interface, the dynamic data source to be migrated out and the dynamic data source to be migrated in are determined; querying the dynamic data source to be migrated out and the dynamic data source to be migrated in to obtain the selected migration out table information and the selected migration in table information, respectively; displaying the selected migration out table information and the selected migration in table information on the preset visual migration task setting interface; in response to the user's second type of selection operation through the visual migration task setting interface, the target migration out table of the dynamic data source to be migrated out and the target migration in table of the dynamic data source to be migrated in are determined; based on the target migration out table and the target migration in table, the corresponding migration task is created; storing the migration task in the static data source.

3. The data migration method of claim 2, wherein, The step of performing data migration among the plurality of dynamic data sources based on the migration configuration information of the static data source comprises: based on the migration configuration parameter and the migration task of the static data source, migrating the data of the target migration out table to the target migration in table.

4. The data migration method of claim 3, wherein, The step of migrating data of the target migration-out table to the target migration-in table based on the migration configuration parameter and the migration task of the static data source comprises: querying target data of the target migration-out table based on the migration configuration parameter and the migration task of the static data source; inserting and updating the target data into the target migration-in table based on a preset insertion and update statement.

5. The data migration method of claim 3, wherein, After the step of migrating data between the plurality of dynamic data sources based on the migration configuration information of the static data source, the method further comprises: performing self-checking on the migration-out table and the migration-in table based on a preset total number of data table checking rule and a table data spot-checking rule to obtain a corresponding self-checking result.

6. A data migration apparatus, characterized by comprising: The data migration device is applied to a data migration system, the data migration system comprising a static data source and a plurality of dynamic data sources, and the data migration device comprising: a migration module configured to migrate data between the plurality of dynamic data sources based on migration configuration information of the static data source, wherein when the migration configuration information is modified, the migration application corresponding to the migration configuration information does not need to be restarted; wherein the data migration device further comprises a concatenation module configured to concatenate migration configuration parameters corresponding to a dynamic data source to be migrated out and a dynamic data source to be migrated in respectively based on a preset resource identifier to obtain a corresponding migration configuration parameter group; storing the migration configuration parameter group in the static data source; wherein the migration configuration information comprises a migration task, and the data migration device further comprises a task module configured to, in a process of executing the migration task, determine whether a migration configuration parameter group corresponding to a resource identifier exists based on a preset aspect judgment rule and the resource identifier corresponding to the migration task; if not, concatenating migration configuration parameters corresponding to a dynamic data source to be migrated out and a dynamic data source to be migrated in respectively designated by the migration task based on a preset pointcut annotation to obtain a corresponding migration configuration parameter group, and storing the migration configuration parameter group in the static data source; storing the resource identifier in a local thread variable of a thread corresponding to the migration task; acquiring the migration configuration parameter group corresponding to the resource identifier in the local thread variable by the thread, and migrating data between the dynamic data source to be migrated out and the dynamic data source to be migrated in based on the migration configuration parameter and the migration task.

7. A terminal device, characterized by comprising: The terminal device comprises a memory, a processor, and a data migration program stored on the memory and executable on the processor, and the data migration program, when executed by the processor, implements the steps of the data migration method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data migration program, and the data migration program, when executed by a processor, implements the steps of the data migration method of any one of claims 1-5.

Images