Data migration method and device

Abstract

The invention provides a data migration method and device, and the method comprises the steps: carrying out the real-time data migration of a table in a business library to a desensitization library through a migration platform, monitoring the table in the business library, and if the table is newly added with a field; judging whether the data corresponding to the field is sensitive data or not; if the field name is sensitive data, a desensitization rule is selected from a rule base to serve as a desensitization processing mode of the sensitive data, the field name and the desensitization rule are stored as field desensitization data, and a notification is sent to a migration platform; the migration platform receives the notification and restarts a migration task, and then the migration platform obtains all desensitization rules and field desensitization data in a rule base through the desensitization processing tool; according to the field desensitization data, the corresponding field in the migration data is desensitized according to the set desensitization rule, and then the data in the service library is migrated to the corresponding table of the desensitization library, so that the data migration efficiency is greatly improved, and the maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a method and device for data migration. Background Art

[0002] Existing users need to migrate data from the business database to a specific database for training the big data platform. However, some data in the business database needs to be desensitized before it can be used for training the big data platform.

[0003] The existing business library is iteratively updated. During the iterative update process, the fields will change. When a new field is added, it will involve whether the data in the field is sensitive data. If it is sensitive data, it needs to be desensitized. At this time, the operation and maintenance personnel will modify the code of the migration platform, business library, and desensitization library, set the desensitization rules in the form of code, and upload it to the migration platform. The migration platform will desensitize the data according to the desensitization rules; restart the migration task corresponding to the migration platform, and the desensitization rules will take effect. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for data migration, which greatly improves the efficiency of migrating data and reduces the cost of maintenance.

[0005] In a first aspect, the present invention provides a method for data migration, comprising the following steps:

[0006] Step 1: Use the migration platform to migrate data from tables in the business database to the desensitization database in real time, monitor the tables in the business database, and if a new field is added to the table, stop the migration task and send a notification to a user;

[0007] Step 2: Determine whether the data corresponding to the field is sensitive data; if not, do not process it; if it is sensitive data, select a desensitization rule from the rule library as the desensitization processing method for the sensitive data, save the field name and the desensitization rule as field desensitization data, create the newly added field in the corresponding table in the desensitization library, and send a notification to the migration platform;

[0008] Step 3. Set up a desensitizing processing tool, which is used to call the set HTTP API to obtain all the desensitizing rules in the rule base and the field desensitized data; the migration platform receives the notification and restarts the migration task, and then the migration platform obtains all the desensitizing rules and the field desensitized data in the rule base through the desensitizing processing tool; according to the field desensitized data, the corresponding fields in the migration data are desensitized according to the set desensitizing rules, and then the data in the business library is started to be migrated to the corresponding table of the desensitizing library.

[0009] In a second aspect, the present invention provides a data migration device, comprising the following modules:

[0010] The monitoring module performs real-time data migration of tables in the business database to the desensitization database through the migration platform, monitors the tables in the business database, and stops the migration task if a new field is added to the table, and sends a notification to a user;

[0011] The setting module determines whether the data corresponding to the field is sensitive data through the user; if it is not sensitive data, it will not be processed; if it is sensitive data, a desensitization rule is selected from the rule library as the desensitization processing method for the sensitive data, the field name and the desensitization rule are saved as field desensitization data, the newly added field is created in the corresponding table in the desensitization library, and a notification is sent to the migration platform;

[0012] The desensitizing migration module sets a desensitizing processing tool, which is used to call the set HTTP API to obtain all the desensitizing rules in the rule base and the field desensitized data; the migration platform receives the notification and restarts the migration task, and then the migration platform obtains all the desensitizing rules and field desensitized data in the rule base through the desensitizing processing tool; according to the field desensitized data, the corresponding fields in the migration data are desensitized according to the set desensitizing rules, and then the data in the business library is migrated to the corresponding table of the desensitizing library.

[0013] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0014] The present invention makes it possible to monitor the data being migrated in real time, stop the migration in time after the data changes, and then quickly set it up to resume the data migration, thereby greatly improving the efficiency of migrating data; and this method does not require too much manual participation or programmer participation during data migration, thereby greatly reducing maintenance costs.

[0015] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0017] Figure 1 is a flow chart of the method in Embodiment 1 of the present invention;

[0018] Figure 2 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0019] The embodiments of the present application provide a method and device for data migration to solve the problem that in the existing data migration process, if there is an increase in fields in the database, it can be quickly processed and the data migration can be quickly resumed.

[0020] The technical solution in the embodiment of the present application has the following general idea:

[0021] 1. Create a DDL monitoring task through the following Flink SQL script and write DDL events to the database_ddl_log table (log table);

[0022] --Create the DB to be monitored

[0023] CREATE TABLE ddl_mysql_table(

[0024] database_name STRING METADATA,

[0025] table_name STRING METADATA,

[0026] op_ts TIMESTAMP_LTZ(3)METADATA,

[0027] ddl STRING,

[0028] binlog_file STRING,

[0029] binlog_pos BIGINT

[0030] )WITH(

[0031] 'connector'='ext-mysql-cdc-ddl',

[0032] 'server-time-zone'='+00:00',

[0033] 'jdbc.properties.useSSL'='false',

[0034] 'hostname'='${ds_cmp_mysql_hostname}',

[0035] 'port'='${ds_cmp_mysql_port}',

[0036] 'username' = '${ds_cmp_mysql_username}',

[0037] 'password' = '${ds_cmp_mysql_password}',

[0038] 'database-name' = '(.+)',

[0039]

[0040] 2. Query the database_ddl_log table every 5 seconds to obtain the data after the execution of the last latestDdlLogId (processing position). Then, use the API provided by Cloudcanal to obtain the list of tasks currently being executed, and obtain the publicHost (public network address) and privateHost (private network address) fields configured for the business database data source from the tasks. Convert them into a Map<String, Task> structure. Use the SQLASTVisitor of the Druid library to analyze the ddl field of the database_ddl_log table to determine whether the DDL statement is of the SQLAlterTableAddColumn type. If so, it indicates that a new field is added. Then, create a DatabaseMosaicRule object and fill in the field name and field type. After collecting the objects into a List, save them to the database, and then call the webhook api provided by DingTalk to send a notification.

[0041] 3. On the desensitization configuration page, create a desensitization rule corresponding to the migration task. First, use the API provided by Cloudcanal to read the task list, select the migration task that needs desensitization configuration, then select the table that needs desensitization configuration under the migration task, and then select the desensitization rule for field configuration and save it.

[0042] 4. The desensitization processing tool implements the CloudCanalProcessorV2 interface of the SDK provided by Cloudcanal. In the class constructor method, use URLConnection to obtain all the table desensitization rules in the task, initialize the corresponding TableProcessor (table processor) for each table. When the data of the table in the migration task changes, the migration platform calls the desensitization processing tool to obtain the TableProcessor (table processor) of the corresponding table, traverse the desensitization rule fields in the table, match the fields with data changes, and if a match is found, fill in the desensitized data according to the desensitization rule by calling the CustomFieldV2.setValue method in the SDK provided by Cloudcanal with encryption or fixed value.

[0043] Existing data migration can be real-time data migration. The data in the business library is constantly updated. After the update, data migration can be performed immediately. Therefore, if there is a situation where a field is added, if the field needs to be desensitized, the present invention can be set up quickly without the need for program participation to complete the desensitization setting and resume data migration.

[0044] Embodiment 1

[0045] like Figure 1 As shown, this embodiment provides a method for data migration, including the following steps:

[0046] Step 1: Use the migration platform to migrate data from tables in the business database to the desensitization database in real time, monitor the tables in the business database, and if a new field is added to the table, stop the migration task and send a notification to a user;

[0047] Step 2: Determine whether the data corresponding to the field is sensitive data; if not, do not process it; if it is sensitive data, select a desensitization rule from the rule library as the desensitization processing method for the sensitive data, save the field name and the desensitization rule as field desensitization data, create the newly added field in the corresponding table in the desensitization library, and send a notification to the migration platform;

[0048] Step 3. Set up a desensitizing processing tool, which is used to call the set HTTP API to obtain all the desensitizing rules in the rule base and the field desensitized data; the migration platform receives the notification and restarts the migration task, and then the migration platform obtains all the desensitizing rules and the field desensitized data in the rule base through the desensitizing processing tool; according to the field desensitized data, the corresponding fields in the migration data are desensitized according to the set desensitizing rules, and then the data in the business library is started to be migrated to the corresponding table of the desensitizing library.

[0049] In this embodiment, preferably, step 1 is specifically as follows: perform real-time data migration of tables in the business library to the desensitizing library through the migration platform, create a DDL monitoring task through the Flink script, write DDL events into the log table, set an interval time to query the log table, obtain the data of the last query of the log table, obtain the currently executed task through the migration platform, obtain the set fields of the business library data configuration from the task, and convert them into a Map structure, use the SQLASTVisitor of the Druid library to analyze the ddl field of the log table, determine whether the DDL statement is of the SQLAlterTableAddColumn type, if so, it indicates that a new field is added, create an object and fill in the field name and field type, and then collect the object into a List and save it in the database, then stop the migration task, and send a notification to a user.

[0050] In this embodiment, preferably, the step 2 is specifically as follows: obtaining all field information of the table from the business library through the migration platform; and obtaining the existing desensitizing rule name from the rule library, generating a configuration page for employees to configure the mapping relationship between field names and desensitizing rules; judging whether the data corresponding to the field is sensitive data; if it is not sensitive data, not selecting the desensitizing rule; if it is sensitive data, selecting a desensitizing rule from the rule library as the desensitizing processing method for the sensitive data, saving the table name, field name and desensitizing rule as field desensitized data, creating the newly added field in the corresponding table in the desensitizing library, and sending a notification to the migration platform.

[0051] In this embodiment, preferably, the step 3 specifically includes: setting a desensitization processing tool, wherein the desensitization processing tool is used to call a set HTTP API to obtain all desensitization rules and field desensitization data in the rule library;

[0052] The migration platform receives the notification and restarts the migration task. Through the setting interface of the SDK provided by the migration platform, the desensitizing processing tool is called to obtain all the desensitizing rules and field desensitized data in the rule library. According to the field desensitized data, the corresponding fields in the migration data are desensitized according to the set desensitizing rules, and the desensitized data is migrated to the corresponding table of the desensitizing library.

[0053] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0054] Embodiment 2

[0055] like Figure 2 As shown, in this embodiment, a data migration device is provided, including the following modules:

[0056] The monitoring module performs real-time data migration of tables in the business database to the desensitization database through the migration platform, monitors the tables in the business database, and stops the migration task if a new field is added to the table, and sends a notification to a user;

[0057] The setting module determines whether the data corresponding to the field is sensitive data through the user; if it is not sensitive data, it will not be processed; if it is sensitive data, a desensitization rule is selected from the rule library as the desensitization processing method for the sensitive data, the field name and the desensitization rule are saved as field desensitization data, the newly added field is created in the corresponding table in the desensitization library, and a notification is sent to the migration platform;

[0058] The desensitizing migration module sets a desensitizing processing tool, which is used to call the set HTTP API to obtain all the desensitizing rules in the rule base and the field desensitized data; the migration platform receives the notification and restarts the migration task, and then the migration platform obtains all the desensitizing rules and field desensitized data in the rule base through the desensitizing processing tool; according to the field desensitized data, the corresponding fields in the migration data are desensitized according to the set desensitizing rules, and then the data in the business library is migrated to the corresponding table of the desensitizing library.

[0059] In this embodiment, preferably, the monitoring module is specifically as follows: performing real-time data migration of tables in the business library to the desensitizing library through the migration platform, creating a DDL monitoring task through the Flink script, writing DDL events into the log table, setting an interval time to query the log table, and obtaining the data of the last query execution of the log table, obtaining the currently executed task through the migration platform, obtaining the set fields of the business library data configuration from the task, and converting them into a Map structure, using the SQLASTVisitor of the Druid library to analyze the ddl field of the log table, judging whether the DDL statement is of the SQLAlterTableAddColumn type, if so, indicating that a new field is added, creating an object and filling in the field name and field type, and then collecting the object into a List and saving it in the database, then stopping the migration task and sending a notification to a user.

[0060] In this embodiment, preferably, the setting module is specifically as follows: obtaining all field information of the table from the business library through the migration platform; and obtaining the existing desensitizing rule name from the rule library, generating a configuration page for employees to configure the mapping relationship between field names and desensitizing rules; judging whether the data corresponding to the field is sensitive data; if it is not sensitive data, not selecting the desensitizing rule; if it is sensitive data, selecting a desensitizing rule from the rule library as the desensitizing processing method for the sensitive data, saving the table name, field name and desensitizing rule as field desensitized data, creating the newly added field in the corresponding table in the desensitizing library, and sending a notification to the migration platform.

[0061] In this embodiment, preferably, the desensitizing migration module is specifically as follows: a desensitizing processing tool is set, and the desensitizing processing tool is used to call the set HTTP API to obtain all the desensitizing rules and field desensitized data in the rule library; the migration platform receives the notification and restarts the migration task, and calls the desensitizing processing tool through the setting interface of the SDK provided by the migration platform to obtain all the desensitizing rules and field desensitized data in the rule library; according to the field desensitized data, the corresponding fields in the migration data are desensitized according to the set desensitizing rules, and the desensitized data is migrated to the corresponding table of the desensitizing library.

[0062] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0063] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for data migration, characterized in that: The steps include: Step 1: Use the migration platform to migrate data from tables in the business database to the desensitization database in real time, monitor the tables in the business database, and if a new field is added to the table, stop the migration task and send a notification to a user; Step 2: Determine whether the data corresponding to the field is sensitive data; If it is not sensitive data, it will not be processed; if it is sensitive data, a desensitization rule will be selected from the rule library as the desensitization processing method for the sensitive data, the field name and the desensitization rule will be saved as field desensitized data, the newly added field will be created in the corresponding table in the desensitization library, and a notification will be sent to the migration platform; Step 3: Set up a desensitization processing tool, which is used to call a set HTTP API to obtain all desensitization rules in the rule library and the field desensitization data; The migration platform receives the notification and restarts the migration task, after which the migration platform obtains all the desensitization rules and field desensitization data in the rule base through the desensitization processing tool; According to the field masking data, the corresponding fields in the migrated data are masked according to the set masking rules, and then the data in the business database is migrated to the corresponding table of the masking database.

2. A data migration method according to claim 1, characterized in that: The step 1 is specifically as follows: perform real-time data migration of the tables in the business library to the desensitizing library through the migration platform, create a DDL monitoring task through the Flink script, write the DDL event into the log table, set the interval time to query the log table, obtain the data of the last query of the log table, obtain the currently executed task through the migration platform, obtain the set field of the business library data configuration from the task, and convert it into a Map structure, use the SQLASTVisitor of the Druid library to analyze the ddl field of the log table, determine whether the DDL statement is of the SQLAlterTableAddColumn type, if so, it indicates that a new field is added, create an object and fill in the field name and field type, then collect the object into a List and save it in the database, then stop the migration task, and send a notification to a user.

3. The method for data migration according to claim 1, characterized in that: The step 2 specifically includes: obtaining all field information of the table from the business library through the migration platform; obtaining the existing desensitization rule name from the rule library, generating a configuration page for employees to configure the mapping relationship between the field name and the desensitization rule; and determining whether the data corresponding to the field is sensitive data; If the data is not sensitive, do not select the desensitization rule; If it is sensitive data, a desensitizing rule is selected from the rule library as the desensitizing processing method for the sensitive data, the table name, field name and desensitizing rule are saved as field desensitized data, the newly added field is created in the corresponding table in the desensitizing library, and a notification is sent to the migration platform.

4. The method for data migration according to claim 1, characterized in that: The step 3 specifically includes: setting a desensitization processing tool, wherein the desensitization processing tool is used to call a set HTTP API to obtain all desensitization rules and field desensitization data in the rule library; The migration platform receives the notification, restarts the migration task, and calls the desensitization processing tool to obtain all desensitization rules and field desensitization data in the rule library through the setting interface of the SDK provided by the migration platform; According to the field masked data, the corresponding fields in the migrated data are masked according to the set masking rules, and the masked data is migrated to the corresponding table in the masking library.

5. A data migration device, characterized in that: Includes the following modules: The monitoring module performs real-time data migration of tables in the business database to the desensitization database through the migration platform, monitors the tables in the business database, and stops the migration task if a new field is added to the table, and sends a notification to a user; The setting module determines whether the data corresponding to the field is sensitive data through the user; If it is not sensitive data, it will not be processed; if it is sensitive data, a desensitization rule will be selected from the rule library as the desensitization processing method for the sensitive data, the field name and the desensitization rule will be saved as field desensitized data, the newly added field will be created in the corresponding table in the desensitization library, and a notification will be sent to the migration platform; The desensitization migration module sets a desensitization processing tool, which is used to call a set HTTP API to obtain all the desensitization rules in the rule library and the field desensitization data; The migration platform receives the notification and restarts the migration task, after which the migration platform obtains all the desensitization rules and field desensitization data in the rule base through the desensitization processing tool; According to the field masking data, the corresponding fields in the migrated data are masked according to the set masking rules, and then the data in the business database is migrated to the corresponding table of the masking database.

6. The data migration device according to claim 5, characterized in that: The monitoring module specifically performs real-time data migration of tables in the business library to the desensitizing library through the migration platform, creates a DDL monitoring task through the Flink script, writes DDL events into the log table, sets an interval to query the log table, obtains the data of the last query of the log table, obtains the currently executed task through the migration platform, obtains the set fields of the business library data configuration from the task, and converts them into a Map structure, uses the SQLASTVisitor of the Druid library to analyze the ddl field of the log table, determines whether the DDL statement is of the SQLAlterTableAddColumn type, and if so, indicates that a new field is added, creates an object and fills in the field name and field type, collects the object into a List and saves it in the database, then stops the migration task and sends a notification to a user.

7. The data migration device according to claim 5, characterized in that: The setting module specifically includes: obtaining all field information of the table from the business library through the migration platform; obtaining the existing desensitization rule name from the rule library, generating a configuration page for employees to configure the mapping relationship between the field name and the desensitization rule; and determining whether the data corresponding to the field is sensitive data; If the data is not sensitive, do not select the desensitization rule; If it is sensitive data, a desensitizing rule is selected from the rule library as the desensitizing processing method for the sensitive data, the table name, field name and desensitizing rule are saved as field desensitized data, the newly added field is created in the corresponding table in the desensitizing library, and a notification is sent to the migration platform.

8. The data migration device according to claim 5, characterized in that: The desensitization migration module specifically includes: setting a desensitization processing tool, which is used to call a set HTTP API to obtain all desensitization rules and field desensitization data in the rule library; The migration platform receives the notification, restarts the migration task, and calls the desensitization processing tool to obtain all desensitization rules and field desensitization data in the rule library through the setting interface of the SDK provided by the migration platform; According to the field masked data, the corresponding fields in the migrated data are masked according to the set masking rules, and the masked data is migrated to the corresponding table in the masking library.

Images