A data de-sensitization method and system based on data assets

Abstract

The present application relates to a kind of data desensitization method and system based on data asset.The data desensitization algorithm corresponding to each field in data is set in advance;When receiving original SQL query statement, the data desensitization algorithm corresponding to the query field in original SQL query statement is parsed;The corresponding data desensitization algorithm is written into original SQL query statement, and the SQL query statement containing desensitization operation is formed;The SQL query statement containing desensitization operation is executed, and the data query result after desensitization is returned according to the corresponding data desensitization algorithm of query field.The present application reduces the error that literal meaning and actual data meaning do not match;Multiple desensitization methods can be coped with multiple scenes.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a data desensitization method and system based on data assets. Background Technology

[0002] With the need for digital transformation in banking, ensuring data security while opening up data is crucial. Traditional solutions typically involve analyzing SQL query result sets, using regular expressions to identify sensitive fields, and then dynamically anonymizing the data by masking these fields. Dynamic masking refers to processing data based on user roles and permissions during data queries, such as replacing certain field values ​​with asterisks or encrypting them. However, traditional techniques rely on matching and recognition, which can lead to significant errors if the literal meaning doesn't match the actual data meaning. Furthermore, regular expression matching of result sets often only allows for uniform masking of common fields, failing to provide tailored masking for different scenarios. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a data anonymization method and system based on data assets. It employs data asset analysis and standardization to reduce errors arising from the mismatch between literal meaning and actual data interpretation. Simultaneously, it utilizes scenario-based configuration of data anonymization algorithms to handle multiple scenarios and multiple anonymization methods. This invention enables one-time inventory checks, one-time benchmarking, and multi-scenario anonymization applications, saving at least half of the labor costs and achieving an anonymization accuracy of over 99%.

[0004] To address the technical problems in existing technologies, this invention proposes a data anonymization method. This method involves pre-setting data anonymization algorithms for each field in the data; upon receiving an original SQL query statement, parsing the data anonymization algorithms corresponding to the query fields in the original SQL query statement; writing the corresponding data anonymization algorithms into the original SQL query statement to form an SQL query statement containing anonymization operations; executing the SQL query statement containing anonymization operations to anonymize the query fields according to the corresponding data anonymization algorithms, and returning the anonymized data query results.

[0005] Preferably, when pre-setting the data anonymization algorithm for each field in the data, the fields in the data are standardized and benchmarked to establish a mapping relationship between the original fields and the standard fields; according to the security level of the standard fields, the corresponding data anonymization algorithm is matched for each standard field, and a mapping relationship between the standard fields and the data anonymization algorithm is established; when the original SQL query statement is received, the original fields in the original SQL query statement are parsed out, and the corresponding data anonymization algorithm is obtained according to the mapping relationship between the original fields and the standard fields, as well as the mapping relationship between the data anonymization algorithms.

[0006] Preferably, the mapping relationship between the original field and the standard field, as well as the mapping relationship between the standard field and the data desensitization algorithm, both exist in the form of a data table.

[0007] Preferably, the system includes three mapping relationship tables: Mapping Relationship Table 1, Mapping Relationship Table 2, and Mapping Relationship Table 3. Mapping Relationship Table 1 stores the mapping relationship between the original fields and the standard fields; Mapping Relationship Table 2 stores the mapping relationship between the standard fields and the data masking algorithm codes; Mapping Relationship Table 3 stores the data masking algorithm codes, as well as the masking function and function parameters corresponding to the data masking algorithm codes. When forming an SQL query statement containing masking operations, the corresponding masking function and function parameters are written into the original SQL query statement.

[0008] Preferably, the first mapping table stores the original field name, the standard field name, and the standard field number; the second mapping table stores the standard field number and the data masking algorithm number. When the original SQL query statement is received and the original fields in the original SQL query statement are parsed out, firstly, the corresponding standard field number is retrieved from the first mapping table based on the original field; then, the corresponding data masking algorithm number is retrieved from the second mapping table based on the standard field number; finally, the corresponding masking function and function parameters are retrieved from the third mapping table based on the data masking algorithm number.

[0009] The present invention also proposes a data anonymization system, comprising: a processor; a database; and a memory, wherein a program is stored, wherein the aforementioned data anonymization method is implemented when the processor executes the program.

[0010] Compared with the prior art, the significant advantages of this invention are:

[0011] (1) Perform metadata analysis and standardization on data asset fields to unify data language and reduce errors.

[0012] (2) Classify and classify standard fields and set different desensitization algorithms for different classifications and sensitivity levels to realize the scenario-based configuration of data desensitization algorithms, which facilitates the desensitization display in multiple scenarios.

[0013] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and claims. Detailed Implementation

[0014] It is readily understood that, based on the technical solution of this invention, various embodiments of this invention can be conceived by those skilled in the art without altering the essential spirit of this invention.

[0015] A data table consists of three parts: the table name, the fields in the table, and the values ​​(data) of the fields. Databases or data warehouses store a large number of data tables generated based on business operations. For commercial banks, these data tables may originate from the same business system or different business systems, collectively constituting the bank's data assets. To fully utilize these data assets, or to meet market, legal, and regulatory needs, it is necessary to query these data assets. However, from a data compliance perspective, commercial banks need to effectively protect data security and privacy. This requires anonymizing the data, rather than providing the raw data to the queryer without any disclosure. Simultaneously, commercial banks also need to ensure the accuracy and reliability of the returned data, which requires that the data anonymization methods be error-free.

[0016] To address this, the present invention proposes a data anonymization method. This method involves pre-setting data anonymization algorithms for each field in the data; upon receiving an original SQL query statement, parsing the data anonymization algorithms corresponding to the query fields in the original SQL query statement; writing the corresponding data anonymization algorithms into the original SQL query statement to form an SQL query statement containing anonymization operations; executing the SQL query statement containing anonymization operations to anonymize the query fields according to the corresponding data anonymization algorithms, and returning the anonymized data query results.

[0017] As a preferred approach, when pre-setting data anonymization algorithms for each field in the data, the fields in the data are standardized and benchmarked to establish a mapping relationship between the original fields and the standard fields. Based on the security level of the standard fields, a corresponding data anonymization algorithm is matched for each standard field, and a mapping relationship is established between the standard fields and the data anonymization algorithms. When the original SQL query statement is received, the original fields in the original SQL query statement are parsed out, and the corresponding data anonymization algorithm is obtained based on the mapping relationship between the original fields and the standard fields, as well as the mapping relationship between the data anonymization algorithms.

[0018] As a preferred approach, the mapping relationship between the original field and the standard field, as well as the mapping relationship between the standard field and the data anonymization algorithm, are both presented in the form of a data table.

[0019] As a preferred method, it includes mapping relationship table one, mapping relationship table two, and mapping relationship table three; wherein, mapping relationship table one stores the mapping relationship between the original fields and the standard fields; mapping relationship table two stores the mapping relationship between the standard fields and the data desensitization algorithm code; mapping relationship table three stores the data desensitization algorithm code, as well as the desensitization function and function parameters corresponding to the data desensitization algorithm code; when forming an SQL query statement containing desensitization operation, the corresponding desensitization function and function parameters are written into the original SQL query statement.

[0020] As a preferred approach, the first mapping table stores the original field names, standard field names, and standard field numbers; the second mapping table stores the standard field numbers and data masking algorithm numbers. When the original SQL query statement is received and the original fields in the original SQL query statement are parsed out, firstly, the corresponding standard field number is retrieved from the first mapping table based on the original field; then, the corresponding data masking algorithm number is retrieved from the second mapping table based on the standard field number; finally, the corresponding masking function and function parameters are retrieved from the third mapping table based on the data masking algorithm number.

[0021] The data anonymization method based on data assets described in this invention will be explained in detail below with reference to a specific embodiment. In practice, to facilitate the storage of calculation results obtained from each step, a series of tables will be created in the calculation system to store the result data of each step. Of course, in practice, various tools such as text documents can also be used to store the calculation results of each step. As an example, the following series of data tables can be used to store the calculation results of each step during the data anonymization process:

[0022] Table 1. Data Asset List Table DATA_ASSET_DETAIL;

[0023] Table 2 Standard Field Security Level Table STD_ATOM_MANA;

[0024] Table 3 Data anonymization algorithm registration form MASK_ARTIC.

[0025] The table templates constructed above can be pre-placed in the system's storage device. The tables described below are two-dimensional tables in a relational database, where each table contains multiple columns, each column being called a field. Each field corresponds to a standard in the data assets; this standard is a specification for a type of data. For example, a personal customer number is a standard, and this standard exists in multiple tables, such as personal credit card information, personal loan customer credit ledgers, and other similar tables.

[0026] Before implementing the data anonymization method, it is necessary to pre-set the data anonymization algorithm corresponding to each field in the data; in this embodiment, the aforementioned Tables 1 to 3 are constructed to pre-set the data anonymization algorithm corresponding to each field in the data.

[0027] First, construct the data asset inventory table DATA_ASSET_DETAIL

[0028] Data assets are acquired, and the original fields in each table within the data assets are standardized and benchmarked. The mapping information between the original fields and the benchmarked standard fields is stored in the data asset list table. That is, the data asset list table stores the mapping relationship between the original fields and the standard fields; in this embodiment, it is referred to as Mapping Relationship Table 1.

[0029] When constructing the data asset list table DATA_ASSET_DETAIL, the reading device can read the list of all tables in the data assets, store the list information in the storage device, form the data asset list table of the entire data warehouse, and then construct the mapping relationship between the original fields and standard fields in all data assets.

[0030] Table 1 is a partial example of the data asset list table DATA_ASSET_DETAIL.

[0031]

[0032] In the data asset list table DATA_ASSET_DETAIL shown in Table 1

[0033] BELGTO_SCHEMA is the schema to which the table belongs, indicating the database schema to which the table belongs; SDM is the job level, which performs operations such as field standardization, code value standardization, redundant field cleaning, and error field cleaning on the metadata of the source business system.

[0034] TABLE_CODE is the table name.

[0035] TABLE_NAME is the Chinese name of the table.

[0036] COL_CODE is the English name of the field, i.e., the original English name of the field.

[0037] COL_NAME is the Chinese name of the field, i.e., the original Chinese name of the field.

[0038] SYS_CODE is the topic or system code to which the table belongs, indicating the system or topic to which the table belongs. The system code is stored before modeling, and the topic code is stored after modeling. INTS in the table is the system abbreviation.

[0039] SYS_NAME is the name of the topic or system to which the table belongs.

[0040] STD_CODE is the standard English name for the field.

[0041] STD_NAME is the standard Chinese name field.

[0042] STD_NUMBER is the standard field number.

[0043] As shown in Table 1, the mapping relationship between original fields and standard fields in each table of the data assets was established by structuring the data asset inventory table. The corresponding standard field number can be retrieved from the mapping relationship table 1 based on the original field. It can be seen that the original field name and the standard field name are the fundamental elements representing the mapping relationship between the original field and the standard field.

[0044] To unify the data language, this embodiment sets standard field numbers in Table 1. This way, the mapping relationship between the original fields and the standard fields includes the standard field number information, which makes it easy to dynamically and conveniently adjust the standard field information.

[0045] Information such as TABLE_CODE, TABLE_NAME, and SYS_CODE / SYS_NAME enriches the data asset inventory table, facilitating the viewing and maintenance of the mapping relationship between original fields and standard fields. This auxiliary information can be retained or excluded from the data asset inventory table, serving as part of the mapping relationship, depending on specific business needs.

[0046] Second, a standard field security level table is established to store the mapping relationship between standard fields and data anonymization algorithm codes. In this embodiment, it is referred to as Mapping Relationship Table Two.

[0047] When defining standard fields, the standards management personnel refer to the industry standard "Guidelines for Data Security Classification in Financial Data Security" (JR_T 0197) issued by the People's Bank of China, and classify the standard fields (such as mobile phone numbers, document types, and names) and assign sensitivity levels based on actual conditions. Different de-identification algorithms are set for different classifications and sensitivity levels. When designing the database table structure of the application system, the corresponding standard field is filled in for each field in the database design document (PDM). During PDM benchmarking, the PDM file is parsed to obtain the original field information and the mapped standard field information. Finally, the standard field mapping information is written into the data asset list table to form the data asset list. The mapping between the original fields and standard fields in the data assets avoids classifying each field in the data assets one by one, and at the same time minimizes the intrusion into existing heterogeneous systems, achieving mapping benchmarking of existing systems.

[0048] Table 2 is a partial example of the standard field security level table STD_ATOM_MANA.

[0049]

[0050] Table 2, Standard Field Security Level Table STD_ATOM_MANA

[0051] STD_NUMBER is the number of the standard field.

[0052] STD_CODE is the standard English name for the field.

[0053] STD_NAME is the standard Chinese name field.

[0054] STD_TYPE is a standard field type.

[0055] STD_LENGTH is the standard field length.

[0056] STD_SCALE specifies the precision of the standard field.

[0057] STY_LEVEL represents the security level of the standard field.

[0058] ARTIC_NUMBER is the data anonymization algorithm number.

[0059] Table 2 shows the security level and de-identification algorithm number corresponding to the standard fields. The de-identification algorithm number corresponding to security level L1 is empty, indicating that L1 level data does not require de-identification.

[0060] As shown in Table 2, a mapping relationship between standard fields and data anonymization algorithm codes was established by constructing a standard field security level table. The corresponding data anonymization algorithm number can be retrieved from the mapping table based on the standard field number. This demonstrates that scenario-based configuration of data functions enables anonymization display across multiple scenarios. It is evident that the standard field number and the data anonymization algorithm number are the fundamental elements representing the mapping relationship between standard fields and data anonymization algorithm codes.

[0061] To clarify data security anonymization levels and achieve data security grading, this embodiment sets the security level, standard field name, standard field type, and standard field length for standard fields in Table 2. This auxiliary information enriches the standard field security level table and helps in viewing, setting, and maintaining the mapping relationship between standard fields and data anonymization algorithm codes. This auxiliary information can be retained or excluded from the standard field security level table based on specific business needs. Setting a data anonymization algorithm number facilitates upgrades or adjustments to anonymization parameters.

[0062] Third, establish a data anonymization algorithm registration table to register the algorithm functions and their parameters. This establishes a mapping relationship between the data anonymization algorithm code and the specific algorithm function; in this embodiment, this is referred to as Mapping Relationship Table Three.

[0063] Table 3. Data Anonymization Algorithm Registration Form (MASK_ARTIC) (Partial Example)

[0064]

[0065] Table 3 shows the data anonymization algorithm registration form MASK_ARTIC.

[0066] ARTIC_NUMBER is the data anonymization algorithm number.

[0067] ARTIC_STATUS represents the status of the data anonymization algorithm, 0 - valid, 1 - invalid.

[0068] ARTIC_NAME is the Chinese name of the data anonymization algorithm.

[0069] ARTIC_FUNC is the data anonymization algorithm function.

[0070] RETAIN_FIRST is used to retain the leading digits (as a parameter of the algorithm function).

[0071] RETAIN_LAST is used to retain the last few bits (parameters of the algorithm function).

[0072] MASK_CHAR is the masking character (a parameter of the algorithm function).

[0073] Table 3 shows the specific algorithm functions and their parameters corresponding to the data masking algorithm numbers. As can be seen from Table 3, by constructing a data masking algorithm registration table, a mapping relationship between data masking algorithm codes and masking functions is established. The corresponding masking function and its parameters can be retrieved from the mapping relationship table based on the data masking algorithm code, thereby constructing an SQL query statement containing the masking operation. It can be seen that the data masking algorithm number and the masking algorithm function and its parameters are the fundamental elements representing the mapping relationship.

[0074] To ensure unified management of data anonymization algorithms, this embodiment includes information such as the algorithm status and its Chinese name in Table 3. This supplementary information enriches the data anonymization algorithm registration table and facilitates the viewing, setting, and maintenance of the mapping relationship between the algorithm number and the algorithm function. This supplementary information can be retained or omitted from the data anonymization algorithm registration table based on specific business needs.

[0075] Upon receiving the original SQL query, the system parses out the original fields. Based on these fields, it retrieves the corresponding standard field IDs from mapping table one; then, based on the standard field IDs, it retrieves the corresponding data masking algorithm IDs from mapping table two; and finally, based on the data masking algorithm IDs, it retrieves the corresponding masking function and its parameters from mapping table three. The corresponding masking function and its parameters are then written into the original SQL query to form a SQL query containing data masking operations. This SQL query is then executed to mask the query fields according to the corresponding data masking algorithm, and the masked data query results are returned and displayed.

[0076] The following is a specific SQL query example. The identity and other data information used in this example are all hypothetical and are for illustrative purposes only.

[0077] In this embodiment, the data assets include two data tables: one is the customer basic information table (SDM_INTS_CUST_BASE_INFO), and the other is the customer historical mobile phone number table (SDM_INTS_CUST_HIS_MOBILE). The two tables are shown below.

[0078]

[0079]

[0080] When a user wants to retrieve all historical mobile phone numbers of a customer, they send the following raw SQL query statement to the system:

[0081]

[0082] After the system receives the aforementioned original SQL query statement, it uses the SQL parser to parse the SELECT function:

[0083] T1.CUST_NAME, -- Customer Name

[0084] T1.ID_NO, -- ID number

[0085] T2.MOBILE, -- Mobile Number

[0086] The original field names are parsed out, and then the corresponding standard field numbers are retrieved from mapping table one. Based on the standard field numbers, the corresponding data masking algorithm numbers are retrieved from mapping table two. Based on the data masking algorithm numbers, the corresponding masking functions and parameters are retrieved from mapping table three. The corresponding masking functions and parameters are then written into the original SQL query statement to form an SQL query statement containing the masking operation, thus completing the dynamic masking. The SQL query statement containing the masking operation is as follows:

[0087]

[0088] Data retrieved from an SQL query that includes data masking operations, such as:

[0089]

[0090] The present invention also proposes a data anonymization system, comprising: a processor; a database; and a memory, wherein a program is stored, wherein the aforementioned data anonymization method is implemented when the processor executes the program.

[0091] The structure of each table in the above embodiments is only an example. In actual operation, the data items in each column are not necessarily only those shown in the tables of the above embodiments, and may also include other data items.

[0092] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto.

[0093] Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this invention should be included within the protection scope of this invention.

[0094] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as including all features in the exemplary embodiments as essential technical features of the claims of this patent.

[0095] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0096] It should be understood that the devices, modules, units, components, etc., included in the system of one embodiment of the present invention can be adaptively modified to be placed in a different device or system than that embodiment. Different devices, modules, units, or components included in the system of the embodiment can be combined into one device, module, unit, or component, or they can be divided into multiple sub-devices, sub-modules, sub-units, or sub-components.

[0097] The apparatuses, modules, units, or components in the embodiments of the present invention can be implemented in hardware, in software running on one or more processors, or in a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement the embodiments of the present invention. The present invention can also be implemented on a computer program product or computer-readable medium for performing some or all of the methods described herein.

Claims

1. A data de-sensitization method, characterized in that, a data de-sensitization algorithm corresponding to each field in data is pre-set, wherein the data de-sensitization algorithm comprises a de-sensitization function and function parameters; when pre-setting the data de-sensitization algorithm corresponding to each field in data, each field in data is standardized, a mapping relationship between the original field in data and the standard field is established; the standard field is classified and sensitively graded, the security level of each standard field is determined, different data de-sensitization algorithms are matched for each standard field according to the security level of each standard field, a mapping relationship between the standard field and the data de-sensitization algorithm code is established, and a mapping relationship between the de-sensitization algorithm code and the de-sensitization function and function parameters corresponding to the de-sensitization algorithm is established; when receiving an original SQL query statement, the query field in the original SQL query statement is parsed, and the data de-sensitization algorithm corresponding to the query field is queried according to the pre-defined mapping relationship; specifically, when receiving the original SQL query statement, the original field in the original SQL query statement is parsed, the corresponding standard field number is queried according to the mapping relationship between the original field and the standard field; the corresponding data de-sensitization algorithm number is queried according to the mapping relationship between the standard field and the data de-sensitization algorithm code; and the corresponding de-sensitization function and function parameters are queried according to the mapping relationship between the de-sensitization algorithm code and the de-sensitization function and function parameters corresponding to the de-sensitization algorithm; the de-sensitization function and function parameters in the data de-sensitization algorithm are written into the query field part of the original SQL query statement to form an SQL query statement containing a de-sensitization function call; wherein the de-sensitization function call directly de-sensitizes data when executing the query; the SQL query statement containing the de-sensitization function call is executed, the query field is de-sensitized according to the corresponding data de-sensitization algorithm, and the de-sensitized data query result is returned. 2.The data de-sensitization method of claim 1, characterized in that, the mapping relationship between the original field and the standard field, and the mapping relationship between the standard field and the data de-sensitization algorithm, both exist in the form of a data table. 3.The data de-sensitization method of claim 2, characterized in that, it comprises a mapping relationship table one, a mapping relationship table two and a mapping relationship table three; wherein, the mapping relationship table one stores the mapping relationship between the original field and the standard field; the mapping relationship table two stores the mapping relationship between the standard field and the data de-sensitization algorithm code; the mapping relationship table three stores the data de-sensitization algorithm code, and the de-sensitization function and function parameters corresponding to the data de-sensitization algorithm code; when forming the SQL query statement containing the de-sensitization operation, the corresponding de-sensitization function and function parameters are written into the original SQL query statement. 4.The data de-sensitization method of claim 3, characterized in that, the original field name, the standard field name and the standard field number are stored in the mapping relationship table one; the standard field number and the data de-sensitization algorithm number are stored in the mapping relationship table two; when receiving the original SQL query statement and parsing the original field in the original SQL query statement, firstly, the corresponding standard field number is queried in the mapping relationship table one according to the original field; Then, the corresponding data desensitization algorithm number is queried in the mapping relationship table two according to the standard field number; Finally, the corresponding desensitization function and function parameters are queried in the mapping relationship table three according to the data desensitization algorithm number.

5. A data de-identification system, comprising: Comprise: A processor; A database; And a memory, wherein a program is stored, Wherein when the processor executes the program, the data desensitization method of any one of claims 1-4 is implemented.

Images