A sensitive data processing method, system and device based on DataX

Abstract

The application discloses a sensitive data processing method, system and device based on DataX, and belongs to the technical field of computer data processing. The implementation process of the method is as follows: an encryption or desensitization strategy and a sensitive data identification rule are added; the strategy and the rule are added to a corresponding task, and sensitive data identification is performed; encryption or desensitization parameters are added to a configuration file; related parameter information added in the configuration file is parsed; and data is encrypted or desensitized according to the encryption or desensitization parameters. The application solves the problems of sensitive data identification and processing in a data synchronization process, realizes automatic identification, encryption or desensitization of sensitive data in the data synchronization process, improves the security of data, greatly improves processing efficiency in combination with data synchronization, and can quickly meet customer demand.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, specifically to a sensitive data processing method, system, and apparatus based on DataX. Background Technology

[0002] DataX is the open-source version of Alibaba Cloud and is an offline data synchronization tool widely used within the Alibaba Group. DataX enables efficient data synchronization between various heterogeneous data sources, including MySQL, Oracle, HDFS, Hive, and HBase.

[0003] DataX, as a data synchronization framework, abstracts the synchronization of different data sources into Reader plugins that read data from the source data source and Writer plugins that write data to the target. Theoretically, the DataX framework can support data synchronization of any data source type. Furthermore, the DataX plugin system forms an ecosystem; each new data source added enables interoperability with existing data sources.

[0004] However, DataX is just a data synchronization tool with limited functionality. It can only perform basic data synchronization and can only be done through JSON configuration files. It lacks a visual interface and is not very user-friendly.

[0005] In reality, there are many needs to identify, encrypt, or de-identify sensitive data while synchronizing data. DataX alone cannot achieve this. Developing a separate program to identify, encrypt, or de-identify sensitive data would be relatively complex and costly. Summary of the Invention

[0006] The technical objective of this invention is to address the above-mentioned shortcomings by providing a sensitive data processing method, system, and apparatus based on DataX. This solution addresses the problem of sensitive data identification and processing during data synchronization, enabling automatic identification, encryption, or desensitization of sensitive data during the synchronization process, thereby improving data security. Combined with data synchronization, this significantly improves processing efficiency and can quickly meet customer needs.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] A sensitive data processing method based on DataX is described below:

[0009] 1) Add encryption or de-identification strategies and sensitive data identification rules.

[0010] Add encryption or de-identification strategies and sensitive data identification rules through the front-end visual interface to generate the required information, including encryption algorithms, encryption keys, and sensitive data identification regular expressions.

[0011] 2) Add strategies and rules to the corresponding tasks and identify sensitive data.

[0012] When adding a new synchronization task, the data encryption or de-identification strategy and sensitive data identification rules are configured into the relevant task. Sensitive data fields are automatically identified according to the sensitive data identification rules and associated with the corresponding encryption or de-identification strategies.

[0013] 3) Add encryption or de-identification parameters to the configuration file.

[0014] After the newly added sensitive data identification is completed, the relevant interfaces of DataX-Web are called to add the encryption or de-identification related parameter information to the executable JSON configuration file of DataX, so as to realize the automatic generation of the JSON configuration file;

[0015] 4) Parse the relevant parameter information added in the configuration file.

[0016] After the task is started according to the scheduled policy, the DataX-Web executor is invoked to start DataX to execute the corresponding JSON configuration file by executing the command. The relevant encrypted or de-identified parameter configuration information added to the JSON configuration file is parsed by the DataX plugin.

[0017] 5) Encrypt or de-identify the data according to the encryption or de-identification parameters.

[0018] When executing a DataX job, the DataX plugin calls the corresponding encryption or desensitization component to encrypt or desensitize the corresponding data fields based on the encryption or desensitization parameters during the data synchronization process. This achieves encryption or desensitization of sensitive data during the data synchronization process.

[0019] This method adds data encryption or de-identification strategies and sensitive data identification rules through a general visual configuration function. When a new synchronization task is added, the data encryption or de-identification strategies and sensitive data identification rules are configured into the relevant task. Sensitive data fields are automatically identified and associated with the corresponding encryption or de-identification strategies based on the sensitive data identification rules. Then, the relevant parameters are added to the executable JSON configuration file of DataX by calling the DataX-Web interface. After the task is started according to the timed strategy, the DataX-Web executor is called to start DataX to execute the corresponding JSON configuration file by executing commands. DataX parses the relevant parameter configuration added to the JSON configuration file and calls the encryption or de-identification components to encrypt or de-identify the data during the data synchronization process based on the parameters.

[0020] In response to the increasing demands for data integration and processing in diversified business systems, and the growing number of related systems, traditional solutions often involve synchronizing data before identifying, encrypting, or de-identifying sensitive data. This approach suffers from high risks of data leakage and low processing efficiency. Our solution addresses the problem of sensitive data identification and processing during data synchronization, enabling automatic identification, encryption, or de-identification of sensitive data during the synchronization process. This significantly improves processing efficiency and allows for rapid fulfillment of customer needs.

[0021] Preferably, the newly added encryption or desensitization strategies and sensitive data identification rules support encryption or desensitization strategies and sensitive data identification rules including SM2, SM4, AES, RSA, DES, DESede, MD5, SHA1, SHA256, SHA512, masking, and truncation, generate the required encryption algorithms, encryption keys, and desensitization parameter information, support the configuration of sensitive data identification regular expressions for various types of data, and allow testing of strategies and rules.

[0022] Preferably, the steps involve adding strategies and rules to the corresponding tasks and identifying sensitive data.

[0023] The newly added synchronization tasks are executed on a scheduled basis through configuration of execution strategies, and have complete task monitoring, data statistics and log viewing functions. They match and filter sensitive data according to the expressions configured by the sensitive data identification rules and associate them with the corresponding encryption or desensitization strategies.

[0024] Preferably, the encryption or desensitization related parameters include operation type, key, fields, and desensitization parameter information.

[0025] Furthermore, the step of adding encryption or de-identification parameters to the configuration file...

[0026] By modifying the relevant interface logic of DataX-Web, the automatic generation of executable JSON configuration files for DataX is achieved, avoiding manual configuration and making it convenient, fast, and accurate. The executor of DataX-Web supports multi-node deployment and unified scheduling of DataX, ensuring the performance and reliability of task execution.

[0027] Furthermore, the relevant parameter information added to the parsed configuration file,

[0028] By modifying the DataX plugin-rdbms-util, the executable JSON configuration file of DataX can support rich parameter configuration, including operation type, key, fields, and de-identification parameter information. It can be extended according to needs to meet different business requirements.

[0029] Furthermore, the data is encrypted or desensitized according to the encryption or desensitization parameters.

[0030] By modifying the DataX plugin-rdbms-util, data encryption or desensitization can be achieved during data synchronization based on operation type, key, fields, and desensitization parameter information. It also supports efficient data synchronization between various heterogeneous data sources, such as files and multiple database types, improving processing efficiency and quickly meeting customer needs.

[0031] Preferably, this method enables efficient, timely, diverse, and well-monitored identification, encryption, or desensitization of sensitive data during the data synchronization process.

[0032] This invention also claims a sensitive data processing system based on DataX. This system adds data encryption or de-identification strategies and sensitive data identification rules through a general visual configuration function. When a new synchronization task is added, the data encryption or de-identification strategies and sensitive data identification rules are configured into the relevant task. Based on the sensitive data identification rules, sensitive data fields are automatically identified and associated with the corresponding encryption or de-identification strategies. Then, by calling the DataX-Web interface, relevant parameters are added to the executable JSON configuration file of DataX. After the task is started according to the timing strategy, the DataX-Web executor is called to start DataX to execute the corresponding JSON configuration file by executing commands. DataX parses the relevant parameter configuration added to the JSON configuration file and, based on the parameters, calls encryption or de-identification components to encrypt or de-identify the data during the data synchronization process.

[0033] This system can implement the aforementioned sensitive data processing method based on DataX.

[0034] The present invention also claims a sensitive data processing device based on DataX, comprising at least one memory and at least one processor;

[0035] The at least one memory is used to store a machine-readable program;

[0036] The at least one processor is configured to invoke the machine-readable program, which is capable of executing the aforementioned sensitive data processing method based on DataX.

[0037] Compared with the prior art, the sensitive data processing method, system, and apparatus based on DataX of the present invention have the following advantages:

[0038] This method for sensitive data identification, encryption, or desensitization based on DataX solves the problem of sensitive data identification, encryption, or desensitization during data synchronization. It achieves automatic identification, encryption, or desensitization of sensitive data during data synchronization, greatly improving processing efficiency and quickly meeting customer needs.

[0039] This method can be run independently as a microservice or deployed in a cluster to achieve load balancing and improve processing efficiency. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the implementation process of the sensitive data processing method based on DataX provided in this embodiment of the invention. Detailed Implementation

[0041] This invention provides a sensitive data processing method based on DataX. This method adds data encryption or de-identification strategies and sensitive data identification rules through a general visual configuration function. When a new synchronization task is added, the data encryption or de-identification strategies and sensitive data identification rules are configured into the relevant task. Sensitive data fields are automatically identified and associated with corresponding encryption or de-identification strategies based on the sensitive data identification rules. Then, relevant parameters are added to the DataX executable JSON configuration file by calling the DataX-Web interface. After the task is started according to the timing strategy, the DataX-Web executor is called to start DataX to execute the corresponding JSON configuration file. DataX parses the relevant parameter configuration added to the JSON configuration file and, based on the parameters, calls encryption or de-identification components to encrypt or de-identify the data during the data synchronization process.

[0042] The specific implementation process of this method is as follows:

[0043] 1) Add encryption or de-identification strategies and sensitive data identification rules.

[0044] Add encryption or de-identification strategies and sensitive data identification rules through the front-end visual interface to generate the required information, including encryption algorithms, encryption keys, and sensitive data identification regular expressions.

[0045] 2) Add strategies and rules to the corresponding tasks and identify sensitive data.

[0046] When adding a new synchronization task, the data encryption or de-identification strategy and sensitive data identification rules are configured into the relevant task. Sensitive data fields are automatically identified according to the sensitive data identification rules and associated with the corresponding encryption or de-identification strategies.

[0047] 3) Add encryption or de-identification parameters to the configuration file.

[0048] After the newly added sensitive data identification is completed, the relevant interfaces of DataX-Web are called to add the encryption or desensitization related parameters, including operation type, key, fields, desensitization parameters, etc., to the executable JSON configuration file of DataX, so as to realize the automatic generation of JSON configuration file;

[0049] 4) Parse the relevant parameter information added in the configuration file.

[0050] After the task is started according to the scheduled policy, the DataX-Web executor is invoked to start DataX to execute the corresponding JSON configuration file by executing the command. The DataX plugin parses the JSON configuration file and adds relevant encryption or de-identification parameter configurations, including information such as operation type, key, fields, and de-identification parameters.

[0051] 5) Encrypt or de-identify the data according to the encryption or de-identification parameters.

[0052] When executing a DataX job, the DataX plugin calls the corresponding encryption or desensitization component to encrypt or desensitize the corresponding data fields based on the encryption or desensitization parameters during the data synchronization process. This achieves encryption or desensitization of sensitive data during the data synchronization process.

[0053] The newly added encryption or desensitization strategies and sensitive data identification rules support encryption or desensitization strategies and sensitive data identification rules including SM2, SM4, AES, RSA, DES, DESede, MD5, SHA1, SHA256, SHA512, masking, and truncation. It generates the required encryption algorithms, encryption keys, and desensitization parameter information, supports the configuration of sensitive data identification regular expressions for various types of data, and allows testing of strategies and rules.

[0054] The process involves adding strategies and rules to the corresponding tasks and identifying sensitive data.

[0055] The newly added synchronization tasks are executed on a scheduled basis through configuration of execution strategies, and have complete task monitoring, data statistics and log viewing functions. They match and filter sensitive data according to the expressions configured by the sensitive data identification rules and associate them with the corresponding encryption or desensitization strategies.

[0056] The step involves adding encryption or de-identification parameters to the configuration file.

[0057] By modifying the relevant interface logic of DataX-Web, the automatic generation of executable JSON configuration files for DataX is achieved, avoiding manual configuration and making it convenient, fast, and accurate. The executor of DataX-Web supports multi-node deployment and unified scheduling of DataX, ensuring the performance and reliability of task execution.

[0058] The relevant parameter information added to the parsing configuration file.

[0059] By modifying the DataX plugin-rdbms-util, the executable JSON configuration file of DataX can support rich parameter configuration, including operation type, key, fields, and de-identification parameter information. It can be extended according to needs to meet different business requirements.

[0060] The data is encrypted or desensitized according to the encryption or desensitization parameters.

[0061] By modifying the DataX plugin-rdbms-util, data encryption or desensitization can be achieved during data synchronization based on operation type, key, fields, and desensitization parameter information. It also supports efficient data synchronization between various heterogeneous data sources, such as files and multiple database types, improving processing efficiency and quickly meeting customer needs.

[0062] This method enables efficient, timely, diverse, and well-monitored identification, encryption, or desensitization of sensitive data during data synchronization.

[0063] In the above case, this method is used as a microservice approach, and a cluster can be deployed to achieve load balancing.

[0064] This method is used as a standalone microservice for the data integration execution engine. Users configure data integration on the web interface, and the data integration execution engine starts up after the integration task is triggered on a scheduled basis or manually.

[0065] This invention also provides a sensitive data processing system based on DataX. This system adds data encryption or de-identification strategies and sensitive data identification rules through a general visual configuration function. When a new synchronization task is added, the data encryption or de-identification strategies and sensitive data identification rules are configured into the relevant task. Sensitive data fields are automatically identified according to the sensitive data identification rules and associated with the corresponding encryption or de-identification strategies. Then, relevant parameters are added to the DataX executable JSON configuration file by calling the DataX-Web interface. After the task is started according to the timing strategy, the DataX-Web executor is called to start DataX to execute the corresponding JSON configuration file by executing commands. DataX parses the relevant parameter configuration added to the JSON configuration file and calls the encryption or de-identification components to encrypt or de-identify the data during the data synchronization process according to the parameters.

[0066] This system can implement the sensitive data processing method based on DataX described in the above embodiments, and the process is as follows:

[0067] 1) Add encryption or de-identification strategies and sensitive data identification rules.

[0068] Add encryption or de-identification strategies and sensitive data identification rules through the front-end visual interface to generate the required information, including encryption algorithms, encryption keys, and sensitive data identification regular expressions.

[0069] 2) Add strategies and rules to the corresponding tasks and identify sensitive data.

[0070] When adding a new synchronization task, the data encryption or de-identification strategy and sensitive data identification rules are configured into the relevant task. Sensitive data fields are automatically identified according to the sensitive data identification rules and associated with the corresponding encryption or de-identification strategies.

[0071] 3) Add encryption or de-identification parameters to the configuration file.

[0072] After the newly added sensitive data identification is completed, the relevant interfaces of DataX-Web are called to add the encryption or desensitization related parameters, including operation type, key, fields, desensitization parameters, etc., to the executable JSON configuration file of DataX, so as to realize the automatic generation of JSON configuration file;

[0073] 4) Parse the relevant parameter information added in the configuration file.

[0074] After the task is started according to the scheduled policy, the DataX-Web executor is invoked to start DataX to execute the corresponding JSON configuration file by executing the command. The DataX plugin parses the JSON configuration file and adds relevant encryption or de-identification parameter configurations, including information such as operation type, key, fields, and de-identification parameters.

[0075] 5) Encrypt or de-identify the data according to the encryption or de-identification parameters.

[0076] When executing a DataX job, the DataX plugin calls the corresponding encryption or desensitization component to encrypt or desensitize the corresponding data fields based on the encryption or desensitization parameters during the data synchronization process. This achieves encryption or desensitization of sensitive data during the data synchronization process.

[0077] The newly added encryption or desensitization strategies and sensitive data identification rules support encryption or desensitization strategies and sensitive data identification rules including SM2, SM4, AES, RSA, DES, DESede, MD5, SHA1, SHA256, SHA512, masking, and truncation. It generates the required encryption algorithms, encryption keys, and desensitization parameter information, supports the configuration of sensitive data identification regular expressions for various types of data, and allows testing of strategies and rules.

[0078] The process involves adding strategies and rules to the corresponding tasks and identifying sensitive data.

[0079] The newly added synchronization tasks are executed on a scheduled basis through configuration of execution strategies, and have complete task monitoring, data statistics and log viewing functions. They match and filter sensitive data according to the expressions configured by the sensitive data identification rules and associate them with the corresponding encryption or desensitization strategies.

[0080] The step involves adding encryption or de-identification parameters to the configuration file.

[0081] By modifying the relevant interface logic of DataX-Web, the automatic generation of executable JSON configuration files for DataX is achieved, avoiding manual configuration and making it convenient, fast, and accurate. The executor of DataX-Web supports multi-node deployment and unified scheduling of DataX, ensuring the performance and reliability of task execution.

[0082] The relevant parameter information added to the parsing configuration file.

[0083] By modifying the DataX plugin-rdbms-util, the executable JSON configuration file of DataX can support rich parameter configuration, including operation type, key, fields, and de-identification parameter information. It can be extended according to needs to meet different business requirements.

[0084] The data is encrypted or desensitized according to the encryption or desensitization parameters.

[0085] By modifying the DataX plugin-rdbms-util, data encryption or desensitization can be achieved during data synchronization based on operation type, key, fields, and desensitization parameter information. It also supports efficient data synchronization between various heterogeneous data sources, such as files and multiple database types, improving processing efficiency and quickly meeting customer needs.

[0086] This invention also provides a sensitive data processing device based on DataX, including at least one memory and at least one processor;

[0087] The at least one memory is used to store a machine-readable program;

[0088] The at least one processor is configured to invoke the machine-readable program, which is capable of executing the sensitive data processing method based on DataX described in the above embodiments.

[0089] Through the specific embodiments described above, those skilled in the art can easily implement the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. Based on the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features to achieve different technical solutions.

[0090] Except for the technical features described in the specification, all other technologies are known to those skilled in the art.

Claims

1. A sensitive data processing method based on DataX, characterized in that, The implementation process of this method is as follows: 1) Added encryption or de-identification strategies and sensitive data identification rules, specifically including: Add encryption or de-identification strategies and sensitive data identification rules through the front-end visual interface to generate the required information, including encryption algorithms, encryption keys, and sensitive data identification regular expressions. 2) Add strategies and rules to the corresponding tasks and perform sensitive data identification, specifically including: When adding a new synchronization task, the data encryption or de-identification strategy and sensitive data identification rules are configured into the relevant task. Sensitive data fields are automatically identified according to the sensitive data identification rules and associated with the corresponding encryption or de-identification strategies. 3) Add encryption or de-identification parameters to the configuration file, specifically including: After the newly added sensitive data identification is completed, the relevant interfaces of DataX-Web are called to add the encryption or de-identification related parameter information to the executable JSON configuration file of DataX, so as to realize the automatic generation of the JSON configuration file; 4) Parse the relevant parameter information added in the configuration file, specifically including: After the task is started according to the scheduled policy, the DataX-Web executor is invoked to start DataX to execute the corresponding JSON configuration file by executing the command. The relevant encrypted or de-identified parameter configuration information added to the JSON configuration file is parsed by the DataX plugin. 5) Encrypt or de-identify the data according to the encryption or de-identification parameters, specifically including: When executing a DataX job, the DataX plugin calls the corresponding encryption or desensitization component to encrypt or desensitize the corresponding data fields based on the encryption or desensitization parameters during the data synchronization process. This achieves encryption or desensitization of sensitive data during the data synchronization process.

2. The sensitive data processing method based on DataX according to claim 1, characterized in that, The newly added encryption or de-identification strategies and sensitive data identification rules support encryption or de-identification strategies and sensitive data identification rules including SM2, SM4, AES, RSA, DES, DESede, MD5, SHA1, SHA256, SHA512, masking, and truncation. It generates the required encryption algorithms, encryption keys, and de-identification parameter information, supports the configuration of sensitive data identification regular expressions for various types of data, and tests the strategies and rules.

3. A sensitive data processing method based on DataX according to claim 1 or 2, characterized in that, The process of adding strategies and rules to the corresponding tasks and identifying sensitive data specifically includes: The newly added synchronization tasks are executed on a scheduled basis through configuration of execution strategies, and have complete task monitoring, data statistics and log viewing functions. They match and filter sensitive data according to the expressions configured by the sensitive data identification rules and associate them with the corresponding encryption or desensitization strategies.

4. The sensitive data processing method based on DataX according to claim 1, characterized in that, The encryption or desensitization related parameters include operation type, key, fields, and desensitization parameter information.

5. A sensitive data processing method based on DataX according to claim 1 or 4, characterized in that, Adding encryption or desensitization parameters to the configuration file also includes: By modifying the relevant interface logic of DataX-Web, the automaton generation of executable JSON configuration files for DataX is achieved. The executor of DataX-Web supports multi-node deployment and unified scheduling of DataX.

6. A sensitive data processing method based on DataX according to claim 5, characterized in that, The related parameter information added to the parsing configuration file also includes: By modifying the plugin-rdbms-util of DataX, the executable JSON configuration file of DataX supports rich parameter configuration, including operation type, key, fields, and de-identification parameter information. It can be extended according to needs to meet different business requirements.

7. A sensitive data processing method based on DataX according to claim 6, characterized in that, The process of encrypting or desensitizing data according to encryption or desensitization parameters also includes: By modifying the DataX plugin-rdbms-util, data encryption or desensitization is achieved during data synchronization based on operation type, key, fields, and desensitization parameter information. It also supports efficient data synchronization between files and various heterogeneous data sources of multiple database types.

8. A sensitive data processing method based on DataX according to claim 1, characterized in that, This method enables efficient, timely, diverse, and well-monitored identification, encryption, or desensitization of sensitive data during data synchronization.

9. A sensitive data processing system based on DataX, characterized in that, The system adds data encryption or desensitization strategies and sensitive data identification rules through a general visual configuration function. When a new synchronization task is added, the data encryption or desensitization strategy and sensitive data identification rules are configured into the relevant task. Based on the sensitive data identification rules, the system automatically identifies sensitive data fields and associates them with the corresponding encryption or desensitization strategies. Then, by calling the DataX-Web interface, the relevant parameters are added to the executable JSON configuration file of DataX. After the task is started according to the timed strategy, the DataX-Web executor is called to start DataX to execute the corresponding JSON configuration file by executing the command. DataX parses the relevant parameter configuration added to the JSON configuration file and calls the encryption or desensitization component to encrypt or desensitize the data during the data synchronization process according to the parameters. The system is capable of implementing the sensitive data processing method based on DataX as described in any one of claims 1 to 8.

10. A sensitive data processing device based on DataX, characterized in that, Includes at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program, which is capable of executing the method according to any one of claims 1 to 8.

Images