Real-time data desensitization system

Abstract

The invention particularly relates to a real-time data desensitization system. According to the data real-time desensitization system, a Java Web technology stack is adopted, a task scheduling module is used for performing project development based on a Springboot framework and is responsible for creating a real-time desensitization task, and a data transmission module is called through a timed task; the data desensitization module is responsible for desensitizing original data by using a desensitization algorithm model; the data transmission module is responsible for extracting original data from source end database information and sending the original data to the data desensitization module; and writing the desensitized data into a target end database to realize real-time data desensitization processing. And the log monitoring module is responsible for monitoring and recording the whole life cycle of the real-time desensitization task from creation to completion in real time. According to the data real-time desensitization system, real-time desensitization processing of multiple data sources is achieved, seamless conversion between heterogeneous data sources is supported, the database safety management process is optimized, supervision on sensitive data is more convenient and efficient, and information safety is further guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and in particular to a real-time data desensitization system. Background Art

[0002] In the past, the rapid development of big data and cloud computing technology has greatly promoted the progress of society. However, the data security issues that follow have also become the focus of people's attention. In particular, sensitive data, such as personal privacy information, is very likely to be leaked if it is not properly handled during transmission. This risk may not only lead to the leakage of personal privacy, but also cause a series of serious social problems.

[0003] Although traditional static data desensitization methods can protect data security to a certain extent, they are powerless when faced with the need to process real-time data streams. This is because traditional desensitization methods can only process data in a static state, and cannot effectively desensitize real-time streaming data. This undoubtedly increases the risk of data transmission.

[0004] Especially in a multi-source heterogeneous data environment, how to efficiently and accurately perform real-time desensitization of sensitive data has become a technical problem that needs to be solved urgently. A multi-source heterogeneous data environment means that the format, structure, and source of the data may be different, which puts higher requirements on data desensitization technology. It needs to be able to quickly and accurately identify sensitive data in various complex data environments and perform effective desensitization.

[0005] In order to adapt to the data processing needs of the big data and cloud computing era, solve the problem of dynamic desensitization processing of sensitive information in real-time data streams in a multi-source heterogeneous data environment, and ensure the security and compliance of data transmission, the present invention proposes a real-time data desensitization system. Summary of the invention

[0006] In order to make up for the deficiencies of the prior art, the present invention provides a simple and efficient real-time data desensitization system.

[0007] The present invention is achieved through the following technical solutions:

[0008] A real-time data desensitization system, which uses Java Web technology stack and is developed based on Springboot framework; it includes data desensitization module, task scheduling module, data transmission module and log monitoring module;

[0009] The task scheduling module is based on the xxl-job component, which is responsible for creating real-time desensitization tasks to the task registration center, providing task scheduling and execution functions, setting a timed cycle start, and calling the data transmission module through the timed task;

[0010] The data desensitization module is responsible for desensitizing the original data using the desensitization algorithm model according to the algorithm type configured in the real-time desensitization task;

[0011] The data transmission module is based on the DataX component and is responsible for extracting the original data using the Reader tool according to the source database information configured in the real-time desensitization task after the real-time desensitization task is started, and sending the extracted original data to the data desensitization module; after calling the data desensitization module to desensitize the original data, the desensitized data is written into the target database to realize real-time data desensitization processing;

[0012] The log monitoring module is responsible for using the APIsix gateway to monitor and record the entire life cycle of the real-time desensitization task from creation to completion, to ensure that each link can be accurately tracked and managed.

[0013] The data desensitization module provides an application interface API or SDK toolkit, and masks, converts or encrypts sensitive information during data transmission based on the SM2 encryption algorithm and the SM4 encryption algorithm.

[0014] The data desensitization module selects a desensitization algorithm model according to the algorithm type configured in the real-time desensitization task:

[0015] If the original data in the real-time desensitization task is a certificate number or a mobile phone number, the desensitization algorithm model uses a custom character replacement technology to mask some of the custom numbers, while custom retaining the necessary information for internal identification in the system;

[0016] If the original data in the real-time desensitization task is address information, the desensitization algorithm model converts the area information into a geographic area identifier through geocoding conversion, and masks the specific street and house number information;

[0017] If the original data in the real-time desensitization task is confidential data, the desensitization algorithm model uses AES (Advanced Encryption Standard) or RSA asymmetric encryption technology to encrypt the confidential data to ensure the security of the data during transmission.

[0018] The task scheduling module deploys a task registration center and configures a task executor, which sends a registration request to the task registration center in real time to ensure real-time processing of desensitized tasks;

[0019] When creating a real-time desensitization task, set the task execution cycle and task parameters and start it; when defining a scheduled task, add the @XxlJob annotation to complete the real-time desensitization task registration; the real-time desensitization task will execute the task periodically according to the set scheduled task time parameters.

[0020] The data transmission module is based on the DataX component to achieve precise docking development between the reader and the writer to ensure smooth data transmission;

[0021] The desensitized data is input into the framework, and the data format conversion is completed according to the target database type; the converted data is output to the writer, and the writer writes the desensitized data into the target database according to the target database information configured in the task.

[0022] The source database and the target database support file data sources, relational databases and big data storage components, and the relational databases include but are not limited to MySQL databases, Oracle databases and SQL Server databases;

[0023] Big data storage components include but are not limited to Hadoop distributed file system, Hbase non-relational database and Hive data warehouse.

[0024] In addition, the data transmission module supports docking with the Elasticsearch search engine.

[0025] The log monitoring module records the conversion, extraction and transmission process of each node in the entire process.

[0026] The log monitoring module captures key performance indicators during task execution, including but not limited to request response time, error rate and traffic pattern;

[0027] The log monitoring module also provides a visual log page to display the recorded data through intuitive charts and statistical data;

[0028] In addition, the log monitoring module also supports the downloading function of log files.

[0029] The real-time desensitization method based on the real-time data desensitization system of the present invention comprises the following steps:

[0030] Step S1, configuring the source data source and the target data source;

[0031] The configuration process automatically completes the adaptation and conversion of heterogeneous data sources, supports database adaptation and conversion between relational databases, between file data sources and relational databases, between big data storage components and relational databases, and between file data sources and big data storage components, corrects the field mapping relationship between the source database and the target database, and completes the preliminary configuration of the source database and the target database;

[0032] Step S2, establishing a real-time desensitization task;

[0033] The task scheduling module is based on the xxl-job component, creates a real-time desensitization task, registers it to the task registration center, sets a scheduled cycle to start, and calls the data transmission module through the scheduled task;

[0034] Step S3, extracting original data;

[0035] After the real-time desensitization task is started, the data transmission module uses the Reader tool to extract the original data according to the source database information configured in the real-time desensitization task, and sends the extracted original data to the data desensitization module;

[0036] Step S4, desensitization treatment;

[0037] The data desensitization module is responsible for desensitizing the original data using the desensitization algorithm model according to the algorithm type configured in the real-time desensitization task;

[0038] Step S5, writing into the target database;

[0039] After calling the data desensitization module to desensitize the original data, the data transmission module completes the data format conversion according to the target database type and writes the desensitized data into the target database;

[0040] Step S6: full-cycle real-time monitoring and log recording;

[0041] The log monitoring module uses the APIsix gateway to monitor and record the entire life cycle of real-time desensitization tasks from creation to completion, ensuring that every link can be accurately tracked and managed;

[0042] Step S7, automatically close the connection;

[0043] After the real-time desensitization task is completed, the connection to the source database and the target database is automatically closed to effectively control the access volume to ensure the stable operation of the system and the security of the data.

[0044] A device for implementing real-time data desensitization, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0045] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0046] The beneficial effects of the present invention are: the real-time data desensitization system realizes real-time desensitization processing of multiple data sources, supports seamless conversion between heterogeneous data sources, optimizes the database security management process, makes the supervision of sensitive data more convenient and efficient, and further ensures information security. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Attached Figure 1 Schematic diagram of the real-time data desensitization system of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] The real-time data desensitization system adopts the Java Web technology stack and is developed based on the Springboot framework. It includes data desensitization modules, task scheduling modules, data transmission modules, and log monitoring modules.

[0051] The task scheduling module is based on the xxl-job component, which is responsible for creating real-time desensitization tasks to the task registration center, providing task scheduling and execution functions, setting a timed cycle start, and calling the data transmission module through the timed task;

[0052] The data desensitization module is responsible for desensitizing the original data using the desensitization algorithm model according to the algorithm type configured in the real-time desensitization task;

[0053] The data transmission module is based on the DataX component and is responsible for extracting the original data using the Reader tool according to the source database information configured in the real-time desensitization task after the real-time desensitization task is started, and sending the extracted original data to the data desensitization module; after calling the data desensitization module to desensitize the original data, the desensitized data is written into the target database to realize real-time data desensitization processing;

[0054] The log monitoring module is responsible for utilizing the high performance and flexibility of the APIsix gateway to monitor and record the entire life cycle of real-time desensitization tasks from creation to completion, to ensure that every link can be accurately tracked and managed.

[0055] The data desensitization module provides an application interface API or SDK toolkit, and masks, converts or encrypts sensitive information during data transmission based on the SM2 encryption algorithm and the SM4 encryption algorithm, so that users can process sensitive data and ensure that the data will not be obtained by unauthorized personnel, thereby ensuring the security and privacy of the data and protecting the privacy and security of users.

[0056] After being packaged as an application interface API or SDK toolkit, developers can easily integrate and use the data desensitization module to ensure data security. The SDK toolkit supports desensitization processing of multiple data types and formats, including text, numbers, dates, etc. At the same time, it also provides flexible configuration options so that developers can adjust according to actual needs.

[0057] The data desensitization module selects a desensitization algorithm model according to the algorithm type configured in the real-time desensitization task:

[0058] If the original data in the real-time desensitization task is a certificate number or a mobile phone number, the desensitization algorithm model uses a custom character replacement technology to mask some of the custom numbers, while custom retaining the necessary information for internal identification within the system, but the original identity information cannot be identified externally. The processing of mobile phone numbers is similar. The middle digits are hidden through masking technology, effectively preventing the direct exposure of the mobile phone number.

[0059] If the original data in the real-time desensitization task is address information, considering the sensitivity of geographic location information, the desensitization algorithm model converts the regional information into a geographic area identifier through geocoding conversion, and masks the specific street and house number information, which not only retains the necessary location information but also avoids the leakage of specific addresses;

[0060] If the original data in the real-time desensitization task is confidential data, the desensitization algorithm model uses AES (Advanced Encryption Standard) or RSA asymmetric encryption technology to encrypt the confidential data to ensure the security of the data during transmission. These encryption technologies can effectively prevent data from being intercepted or tampered with during transmission, ensuring the authenticity and integrity of the data.

[0061] The desensitizing algorithm model can professionally mask and convert common confidential data such as ID card numbers, mobile phone numbers, and home addresses to ensure that these sensitive information is not leaked during transmission or storage.

[0062] The task scheduling module is deeply redeveloped based on the xxl-job framework. It not only inherits the original stability and reliability of the xxl-job component, but also further improves the efficiency and accuracy of task scheduling and execution through customized improvements. Deploy the task registration center and configure the task executor. The task executor sends registration requests to the task registration center in real time to ensure the real-time processing of desensitized tasks.

[0063] When creating a real-time desensitization task, set the task execution cycle and task parameters and start it; when defining a scheduled task, add the @XxlJob annotation to complete the real-time desensitization task registration; the real-time desensitization task will execute the task periodically according to the set scheduled task time parameters.

[0064] The data transmission module is based on the DataX component to achieve precise docking development between the reader and the writer to ensure smooth data transmission;

[0065] The desensitized data is input into the framework, and the data format conversion is completed according to the target database type; the converted data is output to the writer, and the writer writes the desensitized data into the target database according to the target database information configured in the task.

[0066] A wide range of data source types are supported. The source database and the target database support file data sources, relational databases, and big data storage components. The relational databases include but are not limited to Mysql databases, Oracle databases, and SQLServer databases. These databases are widely used in information management systems of various enterprises, providing stable and reliable data storage services.

[0067] In addition to the support of relational databases, the data transmission module has also been extended to the field of big data. Big data storage components include but are not limited to Hadoop distributed file system, Hbase non-relational database and Hive data warehouse.

[0068] In addition, the data transmission module is also optimized for search engines and supports docking with the Elasticsearch search engine. As a search server based on Lucene, the Elasticsearch search engine provides a powerful full-text search function and is widely used in log analysis, full-text retrieval, structured data analysis and other scenarios. Through integration with Elasticsearch, users can quickly retrieve the required information from massive data, greatly improving the availability and value of data.

[0069] The log monitoring module records the conversion, extraction and transmission process of each node in the entire process.

[0070] The log monitoring module captures key performance indicators during task execution, including but not limited to request response time, error rate and traffic pattern;

[0071] These key performance indicator data are not only crucial for immediate problem diagnosis, but also provide valuable reference for long-term performance optimization.

[0072] In order to improve user experience and ease of operation, the log monitoring module also provides a visual log page that displays recorded data through intuitive charts and statistical data, allowing users to easily understand the system status and task execution. Whether for daily monitoring work or for in-depth analysis of system performance, this visual interface is a very useful tool.

[0073] In addition, the log monitoring module also supports the download function of log files. This means that users can export monitoring data for offline analysis or archive for future reference. This function greatly enhances the flexibility of the system, allowing users to conduct more in-depth mining and analysis of log data according to their needs.

[0074] The real-time desensitization method based on the real-time data desensitization system includes the following steps:

[0075] Step S1, configuring the source data source and the target data source;

[0076] The configuration process automatically completes the adaptation and conversion of heterogeneous data sources, supports database adaptation and conversion between relational databases, between file data sources and relational databases, between big data storage components and relational databases, and between file data sources and big data storage components, corrects the field mapping relationship between the source database and the target database, and completes the preliminary configuration of the source database and the target database;

[0077] Step S2, establishing a real-time desensitization task;

[0078] The task scheduling module is based on the xxl-job component, creates a real-time desensitization task, registers it to the task registration center, sets a scheduled cycle to start, and calls the data transmission module through the scheduled task;

[0079] Step S3, extracting original data;

[0080] After the real-time desensitization task is started, the data transmission module uses the Reader tool to extract the original data according to the source database information configured in the real-time desensitization task, and sends the extracted original data to the data desensitization module;

[0081] Step S4, desensitization treatment;

[0082] The data desensitization module is responsible for desensitizing the original data using the desensitization algorithm model according to the algorithm type configured in the real-time desensitization task;

[0083] Step S5, writing into the target database;

[0084] After calling the data desensitization module to desensitize the original data, the data transmission module completes the data format conversion according to the target database type and writes the desensitized data into the target database;

[0085] Step S6: full-cycle real-time monitoring and log recording;

[0086] The log monitoring module uses the APIsix gateway to monitor and record the entire life cycle of real-time desensitization tasks from creation to completion, ensuring that every link can be accurately tracked and managed;

[0087] Step S7, automatically close the connection;

[0088] After the real-time desensitization task is completed, the connection to the source database and the target database is automatically closed to effectively control the access volume to ensure the stable operation of the system and the security of the data.

[0089] The device for implementing real-time data desensitization includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0090] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0091] Compared with the existing technology, this real-time data desensitization system has the following characteristics:

[0092] (1) It realizes the real-time data desensitization function and cooperates with the perfect data security monitoring system to further ensure information security.

[0093] (2) Strengthen measures to make the protection of data privacy and sensitive information more impregnable.

[0094] (3) The database security management process has been optimized, making the supervision of sensitive data more convenient and efficient.

[0095] (4) It realizes desensitization processing of multiple data sources and supports seamless conversion between heterogeneous data sources.

[0096] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time data desensitization system, Features: Use Java Web technology stack and Springboot framework for project development; Including data desensitization module, task scheduling module, data transmission module and log monitoring module; The task scheduling module is based on the xxl-job component, which is responsible for creating real-time desensitization tasks to the task registration center, providing task scheduling and execution functions, setting a timed cycle start, and calling the data transmission module through the timed task; The data desensitization module is responsible for desensitizing the original data using the desensitization algorithm model according to the algorithm type configured in the real-time desensitization task; The data transmission module is based on the DataX component and is responsible for extracting the original data using the Reader tool according to the source database information configured in the real-time desensitization task after the real-time desensitization task is started, and sending the extracted original data to the data desensitization module; After calling the data desensitization module to desensitize the original data, the desensitized data is written into the target database to achieve real-time data desensitization processing; The log monitoring module is responsible for using the APIsix gateway to monitor and record the entire life cycle of the real-time desensitization task from creation to completion, to ensure that each link can be accurately tracked and managed.

2. The real-time data desensitization system according to claim 1, Features: The data desensitization module provides an application interface API or SDK toolkit to mask, convert or encrypt sensitive information during data transmission based on the SM2 encryption algorithm and the SM4 encryption algorithm; The data desensitization module selects a desensitization algorithm model according to the algorithm type configured in the real-time desensitization task: If the original data in the real-time desensitization task is a certificate number or a mobile phone number, the desensitization algorithm model uses a custom character replacement technology to mask some of the custom numbers, while custom retaining the necessary information for internal identification in the system; If the original data in the real-time desensitization task is address information, the desensitization algorithm model converts the area information into a geographic area identifier through geocoding conversion, and masks the specific street and house number information; If the original data in the real-time desensitization task is confidential data, the desensitization algorithm model uses the Advanced Encryption Standard AES or RSA asymmetric encryption technology to encrypt the confidential data to ensure the security of the data during transmission.

3. The real-time data desensitization system according to claim 1, Features: The task scheduling module deploys a task registration center and configures a task executor, which sends a registration request to the task registration center in real time to ensure real-time processing of desensitized tasks; When creating a real-time desensitization task, set the task execution cycle and task parameters and start it; when defining a scheduled task, add the @XxlJob annotation to complete the real-time desensitization task registration; The real-time desensitization task will be executed periodically according to the set scheduled task time parameters.

4. The real-time data desensitization system according to claim 1, Features: The data transmission module is based on the DataX component to achieve precise docking development between the reader and the writer to ensure smooth data transmission; The desensitized data is input into the framework, and the data format conversion is completed according to the target database type; the converted data is output to the writer, and the writer writes the desensitized data into the target database according to the target database information configured in the task.

5. The real-time data desensitization system according to claim 4, Features: The source database and the target database support file data sources, relational databases and big data storage components, and the relational databases include but are not limited to MySQL databases, Oracle databases and SQL Server databases; Big data storage components include but are not limited to Hadoop distributed file system, Hbase non-relational database and Hive data warehouse; In addition, the data transmission module also supports docking with the Elasticsearch search engine.

6. The real-time data desensitization system according to claim 3, Features: The log monitoring module captures and records key performance indicators during the task execution process throughout the life cycle, including but not limited to request response time, error rate and traffic pattern; In addition, the log monitoring module also provides a visual log page, which displays the recorded data through intuitive charts and statistical data; at the same time, the log monitoring module also supports the download function of the log file.

7. A real-time desensitization method based on the real-time data desensitization system according to any one of claims 1 to 6, The following steps are involved: Step S1, configuring the source data source and the target data source; The configuration process automatically completes the adaptation and conversion of heterogeneous data sources, supports database adaptation and conversion between relational databases, between file data sources and relational databases, between big data storage components and relational databases, and between file data sources and big data storage components, corrects the field mapping relationship between the source database and the target database, and completes the preliminary configuration of the source database and the target database; Step S2, establishing a real-time desensitization task; The task scheduling module is based on the xxl-job component, creates a real-time desensitization task, registers it to the task registration center, sets a scheduled cycle to start, and calls the data transmission module through the scheduled task; Step S3, extracting original data; After the real-time desensitization task is started, the data transmission module uses the Reader tool to extract the original data according to the source database information configured in the real-time desensitization task, and sends the extracted original data to the data desensitization module; Step S4, desensitization treatment; The data desensitization module is responsible for desensitizing the original data using the desensitization algorithm model according to the algorithm type configured in the real-time desensitization task; Step S5, writing into the target database; After calling the data desensitization module to desensitize the original data, the data transmission module completes the data format conversion according to the target database type and writes the desensitized data into the target database; Step S6: full cycle real-time monitoring and log recording; The log monitoring module uses the APIsix gateway to monitor and record the entire life cycle of real-time desensitization tasks from creation to completion, ensuring that every link can be accurately tracked and managed; Step S7, automatically close the connection; After the real-time desensitization task is completed, the connection to the source database and the target database is automatically closed to effectively control the access volume to ensure the stable operation of the system and the security of the data.

8. A device for achieving real-time data desensitization, Features: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method as claimed in claim 7 when executing the computer program.

9. A readable storage medium, Features: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to claim 7 is implemented.

Images