Structured data dynamic and static integrated desensitization method

Abstract

The invention relates to the technical field of data desensitization, and discloses a structured data dynamic and static integrated desensitization method which comprises the following steps: receiving a user SQL request and transmitting the SQL request to a database, searching a corresponding return result in the database according to the SQL request, performing dynamic streaming analysis desensitization processing on the return result in a middle layer, and setting a desensitization rule. And adopting dynamic context awareness to assist in dynamic streaming desensitization processing, wherein the dynamic context awareness comprises determination of a current user identity, an access scene and a data consanguinity. According to the method, the database return result is directly subjected to streaming desensitization in the middle layer, an original SQL statement does not need to be modified, grammar analysis vulnerabilities are avoided from the source, the complete desensitization coverage rate is achieved, by combining a dynamic desensitization rule with static desensitization, combination with an ETL tool is supported, when data are extracted to a development and test library, the data are desensitized according to the rule, and the data desensitization efficiency is improved. Intermediate library transition is not needed, and the desensitization effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data desensitization, and in particular to a method for desensitizing structured data in both dynamic and static aspects. Background Art

[0002] Data desensitization refers to the deformation of certain sensitive information through desensitization rules to achieve reliable protection of sensitive privacy data. In this way, desensitized real data sets can be safely used in development, testing, and other non-production environments and outsourced environments. In cases involving customer security data or some commercially sensitive data, real data can be modified and provided for testing without violating system rules. For example, personal information such as ID card numbers, mobile phone numbers, card numbers, and customer numbers all need to be desensitized. This is one of the database security technologies.

[0003] Existing fixed desensitization strategies are not adaptable to the rapid changes in regulations and policies, limiting the flexible adjustment between data processing and privacy protection, resulting in the failure to fully realize the potential of data utilization. In addition, the ever-changing rewriting methods of SQL statements will cause many statements to be bypassed, such as extremely long SQL statements, multi-layer nesting, multi-table associations and other complex statements, thus leading to data leakage. Summary of the Invention

[0004] The present invention provides a method for desensitizing structured data in both dynamic and static modes, which has the beneficial effect of combining dynamic and static desensitization.

[0005] The present invention provides the following technical solution: a method for desensitizing structured data in both dynamic and static modes, comprising:

[0006] Receive user SQL requests and pass them to the database;

[0007] Searching for the corresponding return result in the database according to the SQL request, and performing dynamic streaming desensitization processing on the return result in the middle layer;

[0008] Set desensitization rules;

[0009] Dynamic context awareness is used to assist in dynamic streaming desensitization processing;

[0010] The dynamic context perception includes determining the current user identity, access scenario and data lineage, and adopting dynamic matching desensitization rules to desensitize the returned result data;

[0011] The desensitized return result data is output to the user end.

[0012] As an optional solution to the structured data dynamic and static integrated desensitization method of the present invention, it also includes:

[0013] Determining a data sensitive table based on data lineage inheritance according to the data in the database;

[0014] The desensitized return result data is compared with the data sensitivity table to ensure the consistency of the desensitized expression when the data sensitivity table and the desensitized return result data are copied or modified.

[0015] As an optional solution to the structured data dynamic and static integrated desensitization method of the present invention, it also includes:

[0016] An ETL tool is used to extract data from the database to the database of the requesting end;

[0017] ODBC and JDBC drivers;

[0018] The ODBC and JDBC drivers are used to externally present a virtual database engine, and are combined with ETL tools to implement dynamic and static data desensitization.

[0019] As an optional solution to the structured data dynamic and static integrated desensitization method of the present invention, wherein: the streaming parsing is used to process super-long SQL and complex processing results;

[0020] The streaming parser includes a preprocessing unit, a data preparation unit, and a target parser;

[0021] The pre-processing unit identifies the protocol type of overlong SQL and complex processing results, and matches the corresponding parser;

[0022] The data preparation unit filters the data packets according to the conditions of the parser to select valid data to be parsed;

[0023] The target parser executes parsing and desensitizing rules on the filtered data to assist in obtaining desensitized super-long SQL and complex processing result data.

[0024] As an optional solution to the structured data dynamic and static integrated desensitization method of the present invention, it also includes:

[0025] Data attributes, obtaining the attributes of the data, including the name, description, data type, source, and destination, so as to determine the data attributes;

[0026] Collect data and extract data from the database through the ETL tool;

[0027] Establishing a blood relationship model, determining the type of blood relationship through the attributes of the data, and thereby establishing the blood relationship model;

[0028] Tracking data flow, using an algorithm to track the flow path of the extracted data according to the blood relationship model and the attributes of the data, so as to obtain the blood relationship between the data.

[0029] As an optional solution to the structured data dynamic and static integrated desensitization method of the present invention, the ETL includes:

[0030] Data extraction, obtaining source data in full and / or incremental ways;

[0031] Data conversion, performing format standardization, deduplication and verification of the source data;

[0032] The data is loaded and written into the database of the requesting end according to the preset desensitization rules.

[0033] As an optional solution of the structured data dynamic and static integrated desensitization method of the present invention, it also includes stored procedures and temporary tables;

[0034] By desensitizing the intermediate data in the stored procedure and the temporary table, and establishing a sensitive table through bloodline inheritance, the desensitization consistency of the desensitized stored procedure and the intermediate data in the temporary table when copied or modified is ensured.

[0035] As an optional solution of the method for desensitizing structured data in both dynamic and static modes according to the present invention, it further includes establishing a synonym table, and obtaining the synonym table according to the bloodline inheritance;

[0036] The content of the data is determined by the synonym table, and the desensitization rules are adopted to perform targeted desensitization.

[0037] As an optional solution of the structured data dynamic and static integrated desensitization method of the present invention, it also includes desensitization algorithm, sensitive type and dynamic rule adjustment;

[0038] Using the desensitization algorithm to perform desensitization of the data;

[0039] Formulate the desensitization rule according to the sensitive type;

[0040] The desensitization rules are adaptively modified according to the dynamic rules.

[0041] As an optional solution to the method for desensitizing structured data in both dynamic and static modes according to the present invention, the desensitization rules include that operation and maintenance personnel can only see masked values ​​and auditors can see original values.

[0042] The present invention has the following beneficial effects:

[0043] 1. This method of structured data desensitization, combining both dynamic and static methods, directly performs streaming desensitization on the database return results in the middle layer. The original SQL statements do not need to be modified, thus avoiding syntax parsing vulnerabilities at the root and achieving complete desensitization coverage. By combining dynamic desensitization rules with static desensitization implementation, it supports integration with ETL tools. When extracting data to the development and test libraries, the data has been desensitized according to the rules, eliminating the need for intermediate library transitions and improving the desensitization effect.

[0044] 2. This structured data desensitization method, which integrates dynamic and static data, adopts a streaming processing architecture. It has high efficiency and excellent performance in desensitizing very long SQL statements and complex processing results, and avoids database performance bottlenecks caused by SQL parsing and rewriting.

[0045] 3. This integrated dynamic and static desensitization method for structured data covers the desensitization of objects such as stored procedures, temporary tables, and synonyms, and ensures the consistency of desensitization when data-sensitive tables are copied or modified through a lineage inheritance model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the desensitization method of the present invention.

[0047] Figure 2 This is a UI diagram of the desensitization rule configuration page of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] Example 1

[0050] See also Figure 1-Figure 2 One of the methods for desensitizing structured data in both dynamic and static modes includes:

[0051] Receive user SQL requests and pass them to the database;

[0052] Search for the corresponding return results in the database based on the SQL request, and perform dynamic streaming parsing and desensitization on the return results in the middle layer;

[0053] Set desensitization rules;

[0054] Dynamic context awareness is used to assist in dynamic streaming parsing and desensitization processing;

[0055] Dynamic context awareness includes determining the current user identity, access scenario, and data lineage, and applying dynamic matching desensitization rules to desensitize the returned result data.

[0056] The desensitized return result data is output to the user end.

[0057] Built-in multiple sensitive types and desensitization algorithms, automatically discover sensitive data and configure corresponding desensitization rules according to its sensitive type;

[0058] Flexible multi-mode masking and compliance adaptation are adopted, supporting user-defined masking algorithms and sensitive types (with built-in industry rules), dynamically responding to regulatory changes (such as the requirements of the Personal Information Protection Law), and avoiding the lag of static policies;

[0059] Among them, preferably, the desensitization algorithm uses random mapping or masking and filling;

[0060] Desensitization algorithm, sensitive type and dynamic rule adjustment;

[0061] Use desensitization algorithms to desensitize data;

[0062] Develop desensitization rules based on sensitive types;

[0063] Adaptively modify desensitization rules based on dynamic rules;

[0064] Masking rules include operations personnel only seeing masked values ​​and auditors only seeing original values;

[0065] Existing data masking methods include SQL interception and rewriting at the proxy layer. This intercepts SQL requests at the database proxy layer and rewrites statements, forcing the insertion of masking logic (such as Oracle Data Redaction). However, this solution relies on syntax parsing and can easily miss masking complex SQL statements (such as those with multiple nested columns). Another approach is to embed masking functions in the database, using built-in database functions (such as MySQL's MASK()) to directly process sensitive fields. However, this method requires modifying the data storage structure, is incompatible with heterogeneous databases, and struggles to implement dynamic permission control.

[0066] This instance supports data masking for databases of all types and versions. When a user executes a SQL statement through the middle layer, the SQL statement is not rewritten and the result is obtained from the database as is. The database return result is dynamically stream-masked in the middle layer, and the masked data is returned to the user. Different masking effects (original value, masked value, shielded value, etc.) can be displayed based on the user's identity without changing the actual stored data, achieving 100% dynamic masking for all SQL statements.

[0067] It should be noted that SQL statements include PROCEDURE stored procedures, DBLINK database connections, synonyms, temporary table (TEMPORARY TABLE) desensitization, copy table desensitization (bloodline inheritance), super-long SQL (over 3k bytes) desensitization, prepared statement (prepare) desensitization, processing result desensitization, and support for complex SQL desensitization (multi-table joins, multi-schema joins, deeply nested SQL).

[0068] This ensures compatibility and security without rewriting SQL statements, and performs streaming desensitization on database return results directly in the middle layer. The original SQL statements do not need to be modified, thus avoiding syntax parsing vulnerabilities at the root and achieving 100% desensitization coverage.

[0069] Example 2

[0070] This embodiment is an improvement made on the basis of embodiment 1. For details, please refer to Figure 1-Figure 2 , also includes:

[0071] Determine the data sensitive table based on the data lineage inheritance according to the data in the database;

[0072] Compare the desensitized return result data with the data sensitivity table to ensure the consistency of the desensitized expression between the data sensitivity table and the desensitized return result data when copied or modified;

[0073] Also includes:

[0074] ETL tools, which extract data from the database to the database on the requesting side;

[0075] ODBC and JDBC drivers;

[0076] It is a virtual database engine externally through ODBC and JDBC drivers, and is combined with ETL tools to achieve dynamic and static data desensitization.

[0077] Static desensitization is a security technology for sensitive data. It transforms, replaces, and encrypts original sensitive data (such as personal identity information, financial data, and trade secrets) stored in databases and other data sources according to established rules and algorithms. The result is a set of desensitized data with the same structure as the original data, but with sensitive information hidden or modified.

[0078] Dynamic desensitization by rewriting SQL statements is performed by rewriting SQL statements. When a user executes a statement through the middle layer, the SQL statement containing sensitive data is rewritten. The database layer receives the rewritten data, and desensitization is performed in the database.

[0079] This embodiment improves scenario adaptability through integrated dynamic and static masking. It combines dynamic masking rules with static masking implementation (e.g., user identity, access scenario), and supports integration with ETL tools such as Kettle. For example, when extracting data from a production database to a development or testing database, the data is already masked according to the rules, eliminating the need for intermediate database transitions.

[0080] It is worth noting that the dynamic and static masking is deeply integrated with ETL tools to achieve end-to-end masking from production libraries to test libraries without the need for intermediate storage.

[0081] Among them, Kettle is selected as the ETL tool.

[0082] Streaming parsing is used to process extremely long SQL statements and complex processing results;

[0083] Streaming parsing includes a preprocessing unit, a data preparation unit, and a target parser;

[0084] The pre-processing unit identifies the protocol type of overly long SQL and complex processing results, and matches the corresponding parser;

[0085] The data preparation unit filters the data packets according to the conditions of the parser to select valid data to be parsed;

[0086] The target parser executes parsing and desensitizing rules on the filtered data to assist in obtaining desensitized super-long SQL and complex processing result data‌.

[0087] The streaming processing architecture is adopted, which has high efficiency and excellent performance in desensitizing very long SQL (over 3k bytes) and complex processing results (such as field splicing), and avoids database performance bottlenecks caused by SQL parsing and rewriting.

[0088] Also includes:

[0089] Data attributes, obtain the attributes of the data, including name, description, data type, source and destination, to determine the data attributes;

[0090] Collect data and extract data from the database through ETL tools;

[0091] Establish a blood relationship model, determine the type of blood relationship through the attributes of the data, and thus establish a blood relationship model;

[0092] Tracking data flow: using algorithms to track and extract data flow paths based on the blood relationship model and data attributes, thereby obtaining the blood relationship between data;

[0093] Among them, ETL includes:

[0094] Data extraction, obtaining source data in full and / or incremental ways;

[0095] Data conversion, performing format standardization, deduplication and verification of source data;

[0096] The data is loaded and written into the database of the requesting end according to the preset desensitization rules.

[0097] Example 3

[0098] This embodiment is an improvement made on the basis of embodiment 2. For details, please refer to Figure 1-Figure 2 , also includes stored procedures and temporary tables;

[0099] By desensitizing the intermediate data in stored procedures and temporary tables, and establishing a sensitive table through bloodline inheritance, the desensitization consistency of the desensitized stored procedures and the intermediate data in the temporary tables can be ensured when they are copied or modified.

[0100] Temporary tables are special tables stored in a database system within a temporary folder or a specific database (such as SQL Server's tempdb) for temporary storage of intermediate data. They have a short lifespan and are typically automatically released after the session ends or the transaction commits, but they support regular table operations. Different databases implement them differently. SQL creates them through statements. SQL Server categorizes them into local temporary tables (visible only to the current connection) and global temporary tables (shared by multiple sessions). Oracle distinguishes between session-level (data retained until the end of the session) and transaction-level (cleared after the transaction commits).

[0101] A stored procedure is a set of SQL statements in a database that is used to perform a specific function. It is stored in the database and is permanently valid after being compiled once. Users execute the stored procedure by specifying its name and providing parameters.

[0102] Establish a synonym table and obtain the synonym table based on bloodline inheritance;

[0103] Determine the content of the data through a synonym table, and use desensitization rules for targeted desensitization.

[0104] When copying a data-sensitive table with configured desensitization rules, the desensitization rules will be synchronized to the new table, and the query results of the corresponding fields in the new table will remain in the desensitized state;

[0105] It fully supports complex objects and data lineage inheritance, covers the desensitization of objects such as stored procedures, temporary tables, synonyms, etc., and ensures the desensitization consistency when copying or modifying data-sensitive tables through the lineage inheritance model.

[0106] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0107] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for desensitizing structured data in both dynamic and static modes, characterized in that: include: Receive user SQL requests and pass them to the database; Searching for the corresponding return result in the database according to the SQL request, and performing dynamic streaming parsing and desensitization processing on the return result in the middle layer; Set desensitization rules; Dynamic context awareness is used to assist in dynamic streaming desensitization processing; The dynamic context perception includes determining the current user identity, access scenario and data lineage, and adopting dynamic matching desensitization rules to desensitize the returned result data; The desensitized return result data is output to the user end.

2. The structured data dynamic and static integrated desensitization method according to claim 1 is characterized in that: Also includes: Determining a data sensitive table based on data lineage inheritance according to the data in the database; The desensitized return result data is compared with the data sensitivity table to ensure the consistency of the desensitized expression when the data sensitivity table and the desensitized return result data are copied or modified.

3. The structured data dynamic and static integrated desensitization method according to claim 2, characterized in that: Also includes: ETL tool, which extracts data from the database into the database on the destination end; ODBC and JDBC drivers; The ODBC and JDBC drivers are used to externally present a virtual database engine, and are combined with ETL tools to implement dynamic and static data desensitization.

4. The method for desensitizing structured data in both static and dynamic modes according to any one of claims 1 to 3, characterized in that: The streaming parsing is used to process very long SQL statements and complex processing results; The streaming parser includes a preprocessing unit, a data preparation unit, and a target parser; The pre-processing unit identifies the protocol type of overlong SQL and complex processing results, and matches the corresponding parser; The data preparation unit filters the data packets according to the conditions of the parser to select valid data to be parsed; The target parser executes parsing and desensitizing rules on the filtered data to assist in obtaining desensitized super-long SQL and complex processing result data‌.

5. The structured data dynamic and static integrated desensitization method according to claim 4 is characterized in that: Also includes: Data attributes, obtaining the attributes of the data, including the name, description, data type, source, and destination, so as to determine the data attributes; Collect data and extract data from the database through the ETL tool; Establishing a blood relationship model, determining the type of blood relationship through the attributes of the data, and thereby establishing the blood relationship model; Tracking data flow, using an algorithm to track the flow path of the extracted data according to the blood relationship model and the attributes of the data, so as to obtain the blood relationship between the data.

6. The structured data dynamic and static integrated desensitization method according to claim 5, characterized in that: The ETL includes: Data extraction, obtaining source data in full and / or incremental ways; Data conversion, performing format standardization, deduplication and verification of the source data; The data is loaded and written into the database of the requesting end according to the preset desensitization rules.

7. The structured data dynamic and static integrated desensitization method according to claim 3, characterized in that: Also includes stored procedures and temporary tables; By desensitizing the intermediate data in the stored procedure and the temporary table, and establishing a sensitive table through bloodline inheritance, the desensitization consistency of the desensitized stored procedure and the intermediate data in the temporary table when copied or modified is ensured.

8. The structured data dynamic and static integrated desensitization method according to claim 7, characterized in that: It also includes establishing a synonym table, obtaining the synonym table based on the bloodline inheritance; The content of the data is determined by the synonym table, and the desensitization rules are adopted to perform targeted desensitization.

9. The method for desensitizing structured data in both dynamic and static modes according to any one of claims 1 to 3, characterized in that: It also includes desensitization algorithms, sensitive types, and dynamic rule adjustments; Using the desensitization algorithm to perform desensitization of the data; Formulate the desensitization rule according to the sensitive type; The desensitization rules are adaptively modified according to the dynamic rules.

10. The structured data dynamic and static integrated desensitization method according to claim 9, characterized in that: The desensitization rules include that operation and maintenance personnel can only see masked values ​​and auditors can see original values.

Images