A data desensitization method, device and readable storage medium

Abstract

The application provides a data desensitization method, device and readable storage medium, to solve the problem of low data desensitization efficiency caused by large calculation overhead in the data desensitization process. The method comprises: obtaining an original data table and determining original sensitive data; grouping records in the original data table based on a specific attribute to obtain a grouped data table comprising a plurality of data groups; determining statistical information of the original sensitive data in each data group in the grouped data table and a salt value of each record in the data group to obtain a second intermediate storage table; and desensitizing each original sensitive data using a desensitization formula based on the second intermediate storage table to obtain desensitized data.

Description

Technical Field

[0001] This application relates to the field of information security, and in particular to a method, apparatus and readable storage medium for data desensitization. Background Technology

[0002] With the development of intelligent data analytics, the requirements for data anonymization are becoming increasingly stringent. Intelligent data analytics refers to methods for extracting information from data stored in intelligent data platforms using data analysis tools. Current data anonymization methods mainly include simple masking, hiding, simulation, K-anonymization, and homomorphic encryption. However, while meeting the data anonymization requirements, these methods suffer from high computational costs and cannot provide efficient data analysis services.

[0003] Therefore, there is a lack of a data anonymization method with low computational overhead and high efficiency in the existing technology. Summary of the Invention

[0004] This application provides a method, apparatus, and readable storage medium for data desensitization, which solves the problem of low desensitization efficiency caused by high computational overhead during the data desensitization process.

[0005] In a first aspect, to address the aforementioned problems, this invention provides a data anonymization method, the method comprising:

[0006] Obtain the original data table and identify the original sensitive data; wherein the original data table includes multiple records and each record includes multiple attributes;

[0007] Based on specific attributes, the records in the original data table are grouped to obtain a grouped data table containing multiple data groups;

[0008] By determining the statistical information of the original sensitive data in each data group and the salt value of each record in the data group in the grouped data table, a second intermediate storage table is obtained; wherein, the statistical information indicates the statistical characteristics of the original sensitive data; the statistical information includes the original data group range of the original sensitive data;

[0009] Based on the statistical information and salt value corresponding to each original sensitive data in the second intermediate storage table, the desensitization formula is used to desensitize each original sensitive data to obtain desensitized data.

[0010] Based on the above methods, the statistical information of the original sensitive data is used to perform desensitization using a desensitization formula, thereby reducing computational overhead.

[0011] In one possible implementation, after obtaining the desensitized data, if any desensitized data is not within the original data group of the original sensitive data, then the salt value corresponding to any desensitized data is recalculated, and the original sensitive data corresponding to any desensitized data is re-desensitized using the desensitization formula to obtain second desensitized data; wherein the second desensitized data is within the original data group.

[0012] The above methods ensure that the statistical characteristics of the original sensitive data are within a reasonable range after desensitization, thus ensuring that the desensitized data has analytical capabilities, by determining that the desensitized data is within the range of the original sensitive data.

[0013] In one possible implementation, the method further includes:

[0014] When any of the de-identified data is recovered, a third intermediate storage table, a random key, and a signature key are obtained based on the first authorized terminal and the second authorized terminal; wherein, the random key is used to indicate the number of the data group where the de-identified data is located; the signature key is used to indicate the salt value, the number of the data group, and the statistical information of the record corresponding to the de-identified data;

[0015] Based on the random key of the data group number where any of the de-identified data is located, determine the random number corresponding to any of the de-identified data;

[0016] Based on the random number, determine the first signature key corresponding to any of the de-identified data;

[0017] Based on the third intermediate storage table, determine the first statistical information and the first salt value corresponding to the first signature key;

[0018] Based on the first statistical information and the first salt value in the first signature key, a recovery formula is used to recover any of the de-identified data.

[0019] One possible implementation includes, before restoring any of the de-identified data:

[0020] For each data group number in the second intermediate storage table, a random number is generated, converted into binary format, and a random key is generated.

[0021] A signature key is generated based on the random number corresponding to the data group number in the second intermediate storage table, the salt value corresponding to each record, and statistical information.

[0022] The random key is used to replace the data group number in the second intermediate storage table to obtain the third intermediate storage table;

[0023] The original data table, the second intermediate storage table, and the third intermediate storage table are stored in the first authorized terminal, and the random key and the signature key are stored in the second authorized terminal.

[0024] Using the above method, the original sensitive data, the first intermediate storage table, and the second intermediate storage table are stored in the first authorized terminal; random key K SH Signature key K M The data is stored on a second data terminal, which ensures both the recoverability of the de-identified data and the security of the original sensitive data.

[0025] One possible implementation, wherein grouping records in the original data table based on specific attributes to obtain a grouped data table comprising multiple data groups, includes:

[0026] For the original data table, records with the same attribute value under the specific attribute are grouped into the same data group; the specific attribute is an attribute in the original data table other than the attribute corresponding to the original sensitive data.

[0027] One possible implementation, after determining the statistical information of the original sensitive data in each data group and the salt value of each record in the data group in the grouped data table to obtain the second intermediate storage table, includes:

[0028] To meet user needs, the range of original sensitive data in the statistical information is desensitized using a normalization formula to obtain a new range of data groups.

[0029] Replace the original data group range in the second intermediate storage table with the new data group range.

[0030] Secondly, this invention application provides a data desensitization apparatus, comprising:

[0031] Acquisition Unit: Used to acquire the original data table and determine the original sensitive data; wherein, the original data table includes multiple records and each record includes multiple attributes;

[0032] Grouping unit: Used to group records in the original data table based on specific attributes, to obtain a grouped data table including multiple data groups;

[0033] Determining unit: used to determine the statistical information of the original sensitive data in each data group and the salt value of each record in the data group in the grouped data table, to obtain a second intermediate storage table; wherein, the statistical information indicates the statistical characteristics of the original sensitive data; the statistical information includes the original data group range of the original sensitive data;

[0034] Desensitization unit: used to desensitize each original sensitive data based on the statistical information and salt value corresponding to each original sensitive data in the second intermediate storage table, using a desensitization formula to obtain desensitized data.

[0035] In one possible implementation, the device further includes a checking unit, specifically configured to, if any desensitized data is not within the range of the original data group of the original sensitive data, recalculate the salt value corresponding to any desensitized data, and re-desensitize the original sensitive data corresponding to any desensitized data using the desensitization formula to obtain second desensitized data; wherein the second desensitized data is within the range of the original data group.

[0036] In one possible implementation, the grouping unit is specifically used to group records with the same attribute value under the specific attribute into the same data group for the original data table; the specific attribute is an attribute in the original data table other than the attribute corresponding to the original sensitive data.

[0037] In one possible implementation, the apparatus further includes a recovery unit, specifically configured to, when recovering any of the de-identified data, obtain a third intermediate storage table, a random key, and a signature key based on a first authorized terminal and a second authorized terminal; wherein the random key is used to indicate the number of the data group to which the de-identified data belongs; the signature key is used to indicate the salt value of the record corresponding to the de-identified data, the number of the data group, and statistical information; based on the random key of the number of the data group to which the de-identified data belongs, determine a random number corresponding to the de-identified data; determine a first signature key corresponding to the de-identified data based on the random number; determine a first statistical information and a first salt value corresponding to the first signature key based on the third intermediate storage table; and perform data recovery for the de-identified data using a recovery formula based on the first statistical information and the first salt value in the first signature key.

[0038] In one possible implementation, the recovery unit is further configured to generate a random number for the data group number in the second intermediate storage table, convert it into binary format, and generate a random key; generate a signature key based on the random number corresponding to the data group number in the second intermediate storage table, the salt value corresponding to each record, and statistical information; replace the data group number in the second intermediate storage table with the random key to obtain a third intermediate storage table; store the original data table, the second intermediate storage table, and the third intermediate storage table in the first authorized terminal, and store the random key and the signature key in the second authorized terminal.

[0039] Thirdly, this invention application provides a readable storage medium, comprising,

[0040] memory,

[0041] The memory is used to store instructions that, when executed by a processor, cause an apparatus including the readable storage medium to perform the method as described in the first aspect and any embodiment. Attached Figure Description

[0042] Figure 1 A flowchart of a data anonymization method provided in this application;

[0043] Figure 2 A flowchart of a method for recovering de-identified data provided in this application;

[0044] Figure 3 This is a schematic diagram of a data desensitization device provided in this application. Detailed Implementation

[0045] This application provides a method for desensitizing digital data: by using data statistics and desensitization formulas to desensitize data, the computational cost of data desensitization is reduced and the efficiency of data desensitization is improved.

[0046] The following is a brief introduction to the technical terms described in this application.

[0047] A relational database is a type of database that uses a relational model to organize data, storing it in rows and columns. Rows and columns in a relational database form tables, and a set of tables constitutes the database. Users retrieve data from the database through queries, which are executable codes used to define specific areas within the database. The relational model is a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and the relationships between them. The data in the embodiments of this application all originate from a relational database.

[0048] National cryptographic algorithms: A shorthand for domestically developed cryptographic algorithms, referring to commercially available domestic cryptographic algorithms recognized by the State Cryptography Administration. These mainly include asymmetric algorithms (SM2), hash algorithms (SM3), and symmetric algorithms (SM4).

[0049] Binary: In mathematics and digital circuits, it refers to a base-2 number system, a binary number system that uses two different symbols, 0 (representing zero) and 1 (representing one).

[0050] Adding a salt value: In cryptography, this refers to a method of adding a salt value by inserting a specific string at any fixed position in the password. Adding a salt value can make the hash result different from the hash result using the original password.

[0051] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.

[0052] Please refer to Figure 1 This application provides a method for de-identifying digital data to improve the efficiency of data de-identification. The processing steps of this method are as follows:

[0053] Step 101: Obtain the original data table and identify the original sensitive data.

[0054] It is worth noting that all data sources in this embodiment are relational databases; therefore, the acquired data is presented in data tables. The original data tables mentioned in this step include both numeric and non-numeric data.

[0055] The original data table is a two-dimensional table. A single record representing a fact corresponds to a row in the two-dimensional table. Each record includes multiple attributes or characteristics, corresponding to columns in the two-dimensional table. Therefore, the original data table contains multiple records, and each record includes multiple attributes. Thus, the original data table obtained in this step includes attribute columns C1 to Cn and record rows R1 to Rn. The data in the i-th row and j-th column of the original data table (i.e., the characteristics or attributes of a certain recorded fact) can be denoted as (Ri, Cj).

[0056] Based on the anonymization requirements, certain data in the original data table can be designated as either "original sensitive data" or "original non-sensitive data." "Original sensitive data" and "original non-sensitive data" can refer to one or more attribute columns; that is, "original sensitive data" refers to at least one column of data in the original data table. Therefore, determining the original sensitive data also determines its attributes.

[0057] Step 102: Based on specific attributes, group the records in the original data table to obtain a grouped data table containing multiple data groups.

[0058] Specifically, based on a specific attribute as the grouping criterion, records in the original data table can be divided into at least one data group according to the attribute value of the specific attribute. These data groups are then sequentially numbered with Arabic numerals to obtain data group numbers, such as 01, 02, 03, etc. In the grouped data table, records with the same attribute value under the specific attribute are grouped into the same data group. The specific attribute is an attribute in the original data table other than the attribute corresponding to the original sensitive data. It should be noted that in this embodiment of the application, the original sensitive data is numeric data.

[0059] For example, Table 1 shows the medical records of patients in a hospital in a certain area.

[0060] Table 1

[0061] Hospital Number Patient number age post code Date of visit illness A 001 33 23076 2021.9.10 migraine B 002 23 24785 2021.9.12 tracheitis A 003 36 23077 2021.9.08 asthma B 004 19 23076 2021.8.15 migraine A 005 47 23072 2021.10.8 Fatty liver A 006 12 23076 2021.9.22 influenza C 007 27 23072 2021.9.26 influenza

[0062] Now, assuming that to protect patient privacy in Table 1, we plan to anonymize the attribute values ​​in the age and patient ID attributes, then the hospital ID, postal code, consultation date, and condition attribute values ​​do not need to be anonymized. Therefore, we can use the hospital ID, postal code, consultation date, or condition attribute as the grouping basis. Here, we choose the hospital ID as the grouping basis, which groups records with the same attribute value under the hospital ID attribute. Table 2 shows the grouped data table obtained after grouping the data in Table 1.

[0063] Table 2

[0064]

[0065]

[0066] The grouping results are as follows: records with hospital number A are grouped into one data group, data group number 01; records with hospital number B are grouped into one data group, data group number 02; and records with hospital number C are grouped into one data group, data group number 03.

[0067] Step 103: Determine the statistical information of the original sensitive data in each data group and the salt value of each record in the data group in the grouped data table to obtain the second intermediate storage table; wherein, the statistical information indicates the statistical characteristics of the original sensitive data; the statistical information includes the original data group range of the original sensitive data.

[0068] The following describes the statistical information of the original sensitive data in each data group in the grouped data table and the salt value of each record in the data group.

[0069] First, determine the statistical information of the original sensitive data in each data group in the grouped data table, and save it in the first intermediate storage table.

[0070] Since data analysis usually involves extracting data features based on statistical characteristics, in order for anonymized data to have analytical capabilities, the statistical information of the original sensitive data should be calculated. The statistical information includes count, maximum value, minimum value, sum, arithmetic mean, squared mean, harmonic mean, standard deviation, variance, and checksum.

[0071] Calculate the above statistical information using the original sensitive data in each data group.

[0072] The arithmetic mean, squared mean, harmonic mean, standard deviation, and variance can be calculated using the following formulas.

[0073] Suppose that the data in a dataset are a1, a2, a3, ..., a n ,but

[0074] The arithmetic mean is:

[0075] The squared average is:

[0076] Harmonic mean:

[0077] Standard deviation:

[0078] variance:

[0079] Where n represents the number of data points.

[0080] It should be noted that when storing statistical information, if the sum value is in decimal format, the checksum can be obtained by converting the sum value to hexadecimal after calculating the sum value of the original sensitive data in each data group. In this embodiment, the sum value and checksum do not need to be calculated simultaneously. When the sum value is large or the number of bytes is long, the sum value is replaced by the checksum. The aforementioned operation, by converting the sum value to the checksum, can save space.

[0081] Specifically, given user needs, the original sensitive data consists of attribute values ​​under multiple attributes. Therefore, it's necessary to calculate the data for each attribute value sequentially. For example, to desensitize the age and consultation date attribute values ​​in Table 2, firstly, for the age attribute values, calculate the statistical information for the age attribute values ​​in data groups 01, 02, and 03 respectively. Then, for the consultation date attribute value, calculate the statistical information for the age attribute values ​​in data groups 01, 02, and 03 respectively.

[0082] After calculating the statistical information for each data group, the count (COUNT), maximum value (MAX), minimum value (MIN), sum (SUM), arithmetic mean (AVG), squared mean (QMEAN), harmonic mean (HMEAN), standard deviation (STDEV), variance (VAR), and checksum (CHECKSUM) can be stored in the first intermediate storage table. Table 3 shows the first intermediate storage table.

[0083] Table 3

[0084]

[0085] In this calculation, the maximum value (MAX) and the minimum value (MIN) always appear together. After calculating the sum (SUM), count (COUNT), arithmetic mean (AVG), standard deviation (STDEV), variance (VAR), maximum value (MAX), and minimum value (MIN), depending on the user's needs, if a higher level of desensitization is required, the squared mean (QMEAN) or harmonic mean (HMEAN) is calculated.

[0086] Because the range (MAX, MIN) in the statistical information has a decisive impact on whether the original sensitive data retains its statistical characteristics after anonymization: when the anonymized data is within the original range, it means that the statistical information fluctuates within a reasonable range after anonymization, and the results obtained from the analysis based on the anonymized data are consistent with the results obtained from the analysis based on the original sensitive data, that is, the anonymized data has analytical capabilities. When the original sensitive data is no longer within the range after anonymization, it means that the original sensitive data cannot be statistically analyzed after anonymization and does not have analytical capabilities. Therefore, the consideration of range is consistent throughout the embodiments of this application. When a user makes a request, the original data group range (Min, Max) in the statistical information can be encrypted, and a normalization formula (Min-Max) can be used to obtain the corresponding new data group range (newMin, newMax).

[0087] Specifically, the original data set range (Min, Max) is linearly mapped to the new data set range (newMin, newMax). The normalization formula is as follows:

[0088] newMin = random binary number A (2 or 4 bytes) % 10 + Min

[0089] newMax = random binary number B (2 or 4 bytes) % 100 + Max

[0090] The number of bytes (2 or 4) depends on the system used to process the anonymized data.

[0091] 2. Determine the salt value corresponding to each record in the grouped data table and add it to the first intermediate storage table to obtain the second intermediate storage table.

[0092] The second intermediate storage table includes the data group number of the original sensitive data, the data group statistics, and the salt value of each record in the data group.

[0093] The salt value is a random number generated to protect the data, and its generation is independent of the data content in the corresponding record row. In this embodiment, the number of salt values ​​calculated for each record is 2.

[0094] The formula for calculating salinity is as follows:

[0095] K_S1 = random binary number C (4 bytes) % 2147483647 + 1;

[0096] K_S2 = random binary number D (4 bytes) % 2147483647 + 1;

[0097] Among them, 2147483647 is the maximum value that a single integer data type can represent in a 32-bit system. This minimizes the impact of the salt value on the range of the original sensitive data during the desensitization process.

[0098] After calculating the salt values ​​sequentially, the salt value data is added to the first intermediate storage table to obtain the second intermediate storage table. Table 4 shows a sample table of a data set in the second intermediate storage table.

[0099] Table 4

[0100]

[0101] If the original data group range is encrypted according to user needs during the data anonymization process, the statistical information in the second intermediate storage table will also include the new data group range (newMin, newMax) for each data group.

[0102] The second intermediate storage table includes the batch ID (or K) of the random number corresponding to each original sensitive data. SH The data includes statistics for each data group, raw sensitive data, and salt values. Furthermore, each data group corresponds to at least one record with a K-value. SH The data set includes K_S1, K_S2, and one set of statistical information corresponding to that data set. That is, the statistical information corresponding to the records in each data set is the same.

[0103] Step 104: Based on the statistical information and salt value corresponding to each original sensitive data in the second intermediate storage table, use the desensitization formula to desensitize each original sensitive data to obtain desensitized data.

[0104] Specifically, in this embodiment of the application, after the original sensitive data is desensitized, it must be ensured that the statistical information of the desensitized data can be used for feature analysis, i.e., analysis function; therefore, statistical information should be used to desensitize the original sensitive data. The statistical information can be obtained from the first intermediate storage table.

[0105] For the data (Ri,Cj) in the i-th row and j-th column of the data table, use the following desensitization formula to desensitize the data. The desensitized data is ((Ri,Cj)').

[0106] (R i C j )′=(R i C j )

[0107] +((((K S1 % (SUM)

[0108] +COUNT))%(AVG or QMEAN or HMEAN)∧(MIN or newMIN))

[0109] +StdDev)%K S2 )+(MAX or newMAX)-VAR

[0110] The above formulas include arithmetic and logical operations; where "%" indicates modulo operation; "^" indicates XOR operation; and "or" indicates that one of two or three data items is selected to participate in the operation.

[0111] For the original sensitive data, based on the data group information, statistical information, and salt value corresponding to the original sensitive data contained in the second intermediate storage table described in step 103, the salt value (K) corresponding to the record row where the original sensitive data is located is determined. S1 K S2 The data includes the original sensitive data and the statistical information of the data group where the sensitive data is located. Substituting the above information into the desensitization formula, we obtain the desensitized data.

[0112] Anonymized data can be stored in the target data table. The target data table includes both anonymized data and data from the original data table that does not require anonymization.

[0113] After obtaining the anonymized data, you can also verify whether the anonymized data has analytical capabilities by checking whether it is within the original data set.

[0114] If any desensitized data is not within the original data group of the original sensitive data, then the salt value corresponding to any desensitized data is recalculated, and the original sensitive data corresponding to any desensitized data is re-desensitized using the desensitization formula to obtain the second desensitized data; wherein the second desensitized data is within the original data group.

[0115] Specifically, the range (maximum and minimum values) in the statistical information has a decisive impact on whether the anonymized data possesses analytical capabilities. When the anonymized data falls within the original range, it indicates that the statistical information of the original sensitive data fluctuates within a reasonable range after anonymization. The results obtained from the analysis based on the anonymized data are consistent with the results obtained from the analysis based on the original sensitive data. In other words, the anonymized data can replace the original sensitive data for analysis, meaning the anonymized data possesses analytical capabilities. When the original sensitive data no longer falls within the range after anonymization, it indicates that the original sensitive data cannot be statistically analyzed after anonymization and lacks analytical capabilities. Therefore, the anonymized data must fall within the range of the statistical information corresponding to the original anonymized data. If the anonymized data does not fall within the range of the statistical information corresponding to the original anonymized data, the salt value (K) should be adjusted. S1 K S2 Repeat steps 103-104 until it is determined that the statistical characteristics of the original sensitive data remain unchanged after desensitization, that is, the desensitized data is within the original data group of the original sensitive data. At this point, it can be determined that the desensitized data has analytical functions.

[0116] For example, in a certain original data table, the date attribute is anonymized, and the ID is used as the grouping basis to determine the grouped data table. Table 5 shows a data group in the grouped data table.

[0117] Table 5

[0118] ID Year moon Save date 111 2021 1 4 111 2021 3 12 111 2021 7 7 111 2021 4 7 111 2021 12 8 111 2021 2 1

[0119] The statistical information for the date attribute values ​​stored in this data group is determined as follows: MIN=1, MAX=12, AVG=6.5, SUM=39, COUNT=6, StdDev=3.7283, VAR=13.9, CHECKSUM=27. According to the salt value calculation formula in step 104, the salt value of the first record is determined to be K_S1=1, K_S2=2. For ease of description, the salt value is not calculated for each record here; the salt value of all records in this data group is consistent with that of the first record. Substituting the above statistical information and salt values ​​into the desensitization formula described in step 104, the data in the date attribute column is desensitized, resulting in desensitized data, which is then stored in the target data table. Table 6 shows the target data table obtained after desensitizing the date attribute values ​​in Table 5.

[0120] Table 6

[0121] ID Year moon Save date 111 2021 1 2.8283 111 2021 3 10.8283 111 2021 7 5.8283 111 2021 4 5.8283 111 2021 12 6.8283 111 2021 2 -1.8283

[0122] After data anonymization, based on the data range in the statistical information, it is determined whether the anonymized data has analytical capabilities. The original data group range (MIN=1, MAX=12) is compared with the anonymized data in Table 6, i.e., the saved date attribute values. It is found that -1.8283 is within the original data group range, indicating that the original sensitive data corresponding to -1.8283 needs to be re-anonymized.

[0123] Therefore, by adjusting the salt value corresponding to the record row containing the original sensitive data, the data was re-sensitized. After adjustment, K... S1 =1,K S2 =1. Substituting the above statistical information and the adjusted salt value into the desensitization formula, the desensitized data is 3.8283. Table 7 shows the target data obtained after re-desensitization.

[0124] Table 7

[0125] ID Year moon Save date 111 2021 1 2.8283 111 2021 3 10.8283 111 2021 7 5.8283 111 2021 4 5.8283 111 2021 12 6.8283 111 2021 2 3.8283

[0126] After obtaining Table 7, we re-determine whether the desensitized data is within the range of the original sensitive data group. The result of 3.8283 indicates that the statistical characteristics of the desensitized data are within a reasonable range, meaning that the desensitized data still has analytical functions: the desensitized data can replace the original sensitive data for analysis.

[0127] The data anonymization method provided in this application is reversible; according to user needs, the anonymized data can be selectively restored. Please refer to... Figure 2 The following is a detailed description of the recovery of anonymized data.

[0128] Step 201: When recovering any of the de-identified data, based on the first authorized terminal and the second authorized terminal, obtain the third intermediate storage table, the random key, and the signature key; wherein, the random key is used to indicate the number of the data group where the de-identified data is located; the signature key is used to indicate the salt value of the record corresponding to the de-identified data, the number of the data group, and the statistical information.

[0129] To ensure the security of the original sensitive data during data recovery, a key must be used. This ensures that users of the data platform cannot obtain the original sensitive data through the de-identified data without authorization from their terminals. In this application, data de-identification and data recovery are inversely related. Therefore, during the aforementioned data de-identification steps, a key needs to be generated and stored synchronously to facilitate data recovery after the de-identification operation is completed. The construction of the key can be based on steps 102-104 of the data de-identification process, which will be described in detail below.

[0130] First, based on the second intermediate storage table obtained in step 103, a random number is generated for the data group number in the second intermediate storage table, and then converted into binary format to generate a random key K. SH .

[0131] Specifically, an algorithm or a corresponding hardware random number generator is used to generate a random number (Batch ID) for each data group. The data group number refers to sequentially numbering each data group in the grouped data table using Arabic numerals. For example, the data groups are numbered 01, 02, and 03 respectively. The Batch ID is then encrypted, and the encryption result is converted into binary format, which is the random key K. SH The random key is stored in the random key storage table shown in Table 8. The encryption method for the Batch ID can be a commercial algorithm such as the Chinese national standard symmetric encryption algorithm (SM4) or the Advanced Encryption Standard (AES).

[0132] Table 8

[0133] Batch ID <![CDATA[K SH ]]> … …

[0134] Then, a signature key is generated based on the random number corresponding to the data group number in the second intermediate storage table, the salt value corresponding to each record, and statistical information.

[0135] The signature key is stored in binary format.

[0136] The signature key can be constructed based on the third intermediate storage table.

[0137] The signature key should be stored in the following format:

[0138] K M = (BatchID, K) s1 ,SUM,COUNT,AVG or QMEAN or HMEAN,MIN or newMIN,StdDev,K S2 (MAX or newMAX, VAR)

[0139] Each record has two salt values, and each data group containing an attribute column has corresponding statistics and a random number. Therefore, each piece of data in the original data table corresponds to a signature key K. M K here M The expression includes the Batch ID, salt value, and statistical information of the record row corresponding to the de-identified data. Among these, Batch ID and K... SH The conversion between them can be obtained through binary format conversion.

[0140] It should be noted that, in order to ensure information security, in KM In this dataset, the first item is fixed as a random number, and the statistical information of the data group corresponding to this random number, as well as the random distribution of the salt value of each row of data in the data group, are also included.

[0141] Next, the data group number in the second intermediate storage table is replaced using the random key to obtain the third intermediate storage table. Table 9 shows a sample table of a data group in the third intermediate storage table.

[0142] Table 9

[0143]

[0144] The third intermediate storage table includes the random key corresponding to the random number of the data group, statistical information, and two salt values ​​corresponding to each record in each data group.

[0145] Finally, the original data table, the second intermediate storage table, and the third intermediate storage table are stored in the first authorized terminal, while the random key and the signature key are stored in the second authorized terminal. When a user needs to recover the de-identified data, they must obtain the third intermediate storage table, the random key, and the signature key from the first and second authorized terminals, respectively, to begin the data recovery process.

[0146] Step 202: Determine the random number corresponding to any de-identified data based on the random key of the data group number where any de-identified data is located.

[0147] Since the random key is obtained by converting a random number into binary format, you can get the random key by converting the random key into decimal format.

[0148] Step 203: Determine the first signature key corresponding to any of the de-identified data based on the random number.

[0149] According to step 201, in K M In this process, the first item is fixed as a random number. Therefore, the signature key corresponding to any de-identified data can be determined using the random number.

[0150] Step 204: Based on the third intermediate storage table, determine the first statistical information and the first salt value corresponding to the first signature key.

[0151] Based on the random key, the first data group and the first statistical information corresponding to the de-identified data are determined in the third intermediate storage table.

[0152] Furthermore, by comparing the other data items in the signature key (the statistical information and salt value corresponding to any of the desensitized data) with the first statistical information in the third intermediate storage table, the first salt value corresponding to the sensitive data can be determined.

[0153] By sequentially querying the de-identified data that needs to be recovered, the signature key K corresponding to each piece of de-identified data that needs to be recovered can be determined. M .

[0154] Step 205: Based on the first statistical information and the first salt value in the first signature key, use the recovery formula to recover the data for any of the de-identified data.

[0155] For the elements (R) in the target data table i C j The mathematical restoration formula is as follows:

[0156] (R i C j )=(R i C j )′

[0157] -((((K S1 % (SUM)

[0158] +COUNT))%(AVG or QMEAN or HMEAN)∧(MIN or newMIN))

[0159] +StdDev)%K S2 )-(MAX or newMAX)+VAR

[0160] The recovery formula includes arithmetic and logical operations; where "%" indicates modulo operation and "^" indicates XOR operation.

[0161] Based on the data to be recovered in the target data table, determine the data group containing the data to be recovered one by one, as well as the random number Batch ID corresponding to the data group, and perform data recovery one by one.

[0162] Specifically, the signature key K corresponding to the data is determined based on the Batch ID. M Using K M We can obtain various statistical information, and then, by querying the third intermediate storage table, we can determine the relationship between each item in the statistical information and K. M The correspondence between the values ​​in the table. For example, K. M = (111, 2, 58, 6, 7, -1, 4, 1, 13, 3). First, we can determine that 111 is the Batch ID, so we convert 111 to binary to get K. SH Then, the statistical information corresponding to the data group containing the de-identified data that needs to be recovered is queried in the third intermediate storage table. This statistical information is then compared with the signature key K. M By comparing the data from the second item and subsequent items, K can be determined. MThe statistical information category to which each value belongs is determined. At this point, the signature key still has two values ​​remaining, namely the salt values ​​K_S1 and K_S2. Finally, the statistical information and salt values ​​corresponding to the desensitized data to be recovered can be substituted into the data recovery formula to obtain the original sensitive data.

[0163] This application provides a data anonymization method that uses statistical information from the original sensitive data and an anonymization formula to reduce computational overhead. By ensuring the anonymized data remains within the range of the original sensitive data, the method guarantees its analytical capabilities. Simultaneously, the original sensitive data, along with a first intermediate storage table and a second intermediate storage table, are stored on a first authorized terminal; a random key K is also included. SH Signature key K M The data is stored on a second data terminal, which ensures both the reversibility of the de-identified data and the security of the original sensitive data.

[0164] Based on the same inventive concept, this application provides a waste gas treatment device for use in a China VI emission standard after-treatment system. This device is similar to the aforementioned... Figure 1 The data anonymization method shown corresponds to the specific implementation of this device, which can be found in the description of the aforementioned method embodiments section. Repeated descriptions will not be repeated here. Figure 3 The device includes:

[0165] Acquisition unit 301: used to acquire the original data table and determine the original sensitive data; wherein the original data table includes multiple records and each record includes multiple attributes.

[0166] Specifically, the original data table is a two-dimensional table. A single record or entry representing a specific fact corresponds to a row in the two-dimensional table. Each record includes multiple attributes or characteristics, corresponding to columns in the two-dimensional table. Therefore, the original data table contains multiple records, and each record includes multiple attributes.

[0167] Therefore, the obtained original data table includes attribute columns C1 to Cn and record rows R1 to Rn; then, the data in the i-th row and j-th column of the original data table (i.e., the characteristics or attributes of a certain record fact) can be denoted as (Ri, Cj).

[0168] The original data table includes both sensitive and non-sensitive data, both listed in terms of attribute columns. In other words, sensitive data refers to at least one column of data in the original data table. Therefore, determining the sensitive data also determines its attributes.

[0169] Grouping unit 302: Used to group records in the original data table based on specific attributes to obtain a grouped data table including multiple data groups.

[0170] Specifically, in the grouped data table, records with the same attribute value under a specific attribute are grouped into the same data group. The specific attribute is an attribute in the original data table other than the attribute corresponding to the original sensitive data.

[0171] Determining unit 303: used to determine the statistical information of the original sensitive data in each data group and the salt value of each record in the data group in the grouped data table, to obtain a second intermediate storage table; wherein, the statistical information indicates the statistical characteristics of the original sensitive data; the statistical information includes the original data group range of the original sensitive data.

[0172] Specifically, the statistical information includes count, maximum value, minimum value, sum, arithmetic mean, squared mean, harmonic mean, standard deviation, variance, and checksum.

[0173] Desensitization unit 304: Used to desensitize each original sensitive data based on the statistical information and salt value corresponding to each original sensitive data in the second intermediate storage table, using a desensitization formula to obtain desensitized data.

[0174] Specifically, in this embodiment of the application, after the original sensitive data is desensitized, it must be ensured that the statistical information of the desensitized data can be used for feature analysis, i.e., analysis function; therefore, statistical information should be used to desensitize the original sensitive data. The statistical information can be obtained from the first intermediate storage table.

[0175] For the data (Ri,Cj) in the i-th row and j-th column of the data table, use the following desensitization formula to desensitize the data. The desensitized data is ((Ri,Cj)').

[0176] (R i C j )′=(R i C j )

[0177] +((((K S1 % (SUM)

[0178] +COUNT))%(AVG or QMEAN or HMEAN)∧(MIN or newMIN))

[0179] +StdDev)%K S2 )+(MAX or newMAX)-VAR

[0180] The above formulas include arithmetic and logical operations; where "%" indicates modulo operation; "^" indicates XOR operation; and "or" indicates that one of two or three data items is selected to participate in the operation.

[0181] Specifically, in this embodiment of the application, after the original sensitive data is desensitized, it must be ensured that the statistical information of the desensitized data can be used for feature analysis, i.e., analysis function; therefore, statistical information should be used to desensitize the original sensitive data. The statistical information can be obtained from the first intermediate storage table.

[0182] For the data (Ri,Cj) in the i-th row and j-th column of the data table, use the following desensitization formula to desensitize the data. The desensitized data is ((Ri,Cj)').

[0183] (R i C j )′=(R i C j )

[0184] +((((K S1 % (SUM)

[0185] +COUNT))%(AVG or QMEAN or HMEAN)∧(MIN or newMIN))

[0186] +StdDev)%K S2 )+(MAX or newMAX)-VAR

[0187] The above formulas include arithmetic and logical operations; where "%" indicates modulo operation; "^" indicates XOR operation; and "or" indicates that one of two or three data items is selected to participate in the operation.

[0188] Anonymized data can be stored in the target data table. The target data table includes both anonymized data and data from the original data table that does not require anonymization.

[0189] The data desensitization device further includes a checking unit, specifically used to recalculate the salt value corresponding to any desensitized data if any desensitized data is not within the range of the original data group of the original sensitive data, and to re-desensitize the original sensitive data corresponding to any desensitized data using the desensitization formula to obtain second desensitized data; wherein the second desensitized data is within the range of the original data group.

[0190] The data desensitization device further includes a recovery unit, used to, when recovering any desensitized data, obtain a third intermediate storage table, a random key, and a signature key based on a first authorized terminal and a second authorized terminal; wherein, the random key is used to indicate the number of the data group to which the desensitized data belongs; the signature key is used to indicate the salt value of the record corresponding to the desensitized data, the number of the data group, and statistical information; based on the random key of the number of the data group to which the desensitized data belongs, a random number corresponding to the desensitized data is determined; based on the random number, a first signature key corresponding to the desensitized data is determined; based on the third intermediate storage table, first statistical information and a first salt value corresponding to the first signature key are determined; based on the first statistical information and the first salt value in the first signature key, a recovery formula is used to recover the data for the desensitized data.

[0191] The recovery unit is further configured to generate a random number for the data group number in the second intermediate storage table, convert it into binary format, and generate a random key; generate a signature key based on the random number corresponding to the data group number in the second intermediate storage table, the salt value corresponding to each record, and statistical information; replace the data group number in the second intermediate storage table with the random key to obtain a third intermediate storage table; store the original data table, the second intermediate storage table, and the third intermediate storage table in the first authorized terminal, and store the random key and the signature key in the second authorized terminal.

[0192] Based on the same inventive concept, embodiments of this application also provide a readable storage medium, including:

[0193] memory,

[0194] The memory is used to store instructions that, when executed by a processor, cause the apparatus including the readable storage medium to perform the data desensitization method described above.

[0195] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0196] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0197] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0199] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: Universal Serial Bus flash disks, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0200] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for data de-identification, the method comprising: The method comprises: obtaining an original data table and determining original sensitive data; wherein the original data table comprises a plurality of records and each record comprises a plurality of attributes; grouping the records in the original data table according to a specific attribute to obtain a grouped data table comprising a plurality of data groups; determining statistical information of the original sensitive data in each data group in the grouped data table and a salt value of each record in the data group to obtain a second intermediate storage table; wherein the statistical information indicates statistical characteristics of the original sensitive data; the statistical information comprises an original data group range of the original sensitive data; desensitizing each original sensitive data based on the statistical information and the salt value corresponding to each original sensitive data in the second intermediate storage table using a desensitization formula to obtain desensitized data; if any desensitized data is not within the original data group range of the original sensitive data, recalculating a salt value corresponding to the any desensitized data and desensitizing the original sensitive data corresponding to the any desensitized data using the desensitization formula to obtain second desensitized data; wherein the second desensitized data is within the original data group range.

2. The method of claim 1, wherein, The method further comprises: when the any desensitized data is recovered, obtaining a third intermediate storage table, a random key and a signature key based on a first authorized terminal and a second authorized terminal; wherein the random key is used to indicate a number of a data group in which the any desensitized data is located; the signature key is used to indicate a salt value of a record corresponding to the any desensitized data, a number of a data group and statistical information; determining a random number corresponding to the any desensitized data based on the random key of the number of the data group in which the any desensitized data is located; determining a first signature key corresponding to the any desensitized data according to the random number; determining first statistical information and a first salt value corresponding to the first signature key based on the third intermediate storage table; using a recovery formula to recover the any desensitized data based on the first statistical information and the first salt value in the first signature key.

3. The method of claim 2, wherein, Before the any desensitized data is recovered, the method comprises: generating a random number for the number of the data group in the second intermediate storage table and converting the random number into a binary format to generate a random key; generating a signature key based on the random number corresponding to the number of the data group in the second intermediate storage table, a salt value corresponding to each record and statistical information; replacing the number of the data group in the second intermediate storage table with the random key to obtain a third intermediate storage table; storing the original data table, the second intermediate storage table and the third intermediate storage table in the first authorized terminal and storing the random key and the signature key in the second authorized terminal.

4. The method of claim 1, wherein, The grouping the records in the original data table according to a specific attribute to obtain a grouped data table comprising a plurality of data groups comprises: dividing records with the same attribute value under the specific attribute in the same data group in the original data table.

5. The method of claim 1, wherein, After the determining statistical information of the original sensitive data in each data group in the grouped data table and a salt value of each record in the data group to obtain a second intermediate storage table, the method comprises: According to the user demand, the original data group range where the original sensitive data in the statistical information is located is desensitized using a normalization formula to obtain a new data group range; The original data group range in the second intermediate storage table is replaced with the new data group range.

6. An apparatus for data de-identification, the apparatus comprising: Comprise: An acquisition unit is configured to acquire an original data table and determine original sensitive data; the original data table comprises a plurality of records and each record comprises a plurality of attributes; A grouping unit is configured to group the records in the original data table according to a specific attribute to obtain a grouped data table comprising a plurality of data groups; A determination unit is configured to determine statistical information of the original sensitive data in each data group in the grouped data table and a salt value of each record in the data group to obtain a second intermediate storage table; the statistical information indicates statistical characteristics of the original sensitive data; the statistical information comprises an original data group range of the original sensitive data; A desensitization unit is configured to desensitize each original sensitive data in the second intermediate storage table using a desensitization formula based on the statistical information and the salt value corresponding to the original sensitive data to obtain desensitized data; An inspection unit is configured to, if any desensitized data is not within the original data group range of the original sensitive data, recalculate the salt value corresponding to the any desensitized data and desensitize the original sensitive data corresponding to the any desensitized data using the desensitization formula to obtain second desensitized data; the second desensitized data is within the original data group range.

7. The apparatus of claim 6, wherein, The grouping unit is specifically configured to, for the original data table, divide the records with the same attribute value under the specific attribute in the same data group.

8. The apparatus of claim 6, wherein, The device further comprises a recovery unit configured to, when recovering the any desensitized data, acquire a third intermediate storage table, a random key and a signature key based on a first authorized terminal and a second authorized terminal; the random key is used to indicate the number of the data group where the any desensitized data is located; the signature key is used to indicate the salt value of the record corresponding to the any desensitized data, the number of the data group and the statistical information; determine the random number corresponding to the any desensitized data based on the random key of the number of the data group where the any desensitized data is located; determine the first signature key corresponding to the any desensitized data according to the random number; determine the first statistical information and the first salt value corresponding to the first signature key based on the third intermediate storage table; and perform data recovery on the any desensitized data using a recovery formula based on the first statistical information and the first salt value in the first signature key.

9. The apparatus of claim 8, wherein, The recovery unit is further configured to generate a random number for the number of the data group in the second intermediate storage table and convert it into a binary format to generate a random key; and generate a signature key based on the random number corresponding to the number of the data group in the second intermediate storage table, the salt value of each record and the statistical information. The data group number in the second intermediate storage table is replaced by using the random key to obtain a third intermediate storage table; the original data table, the second intermediate storage table and the third intermediate storage table are saved in the first authorized terminal, and the random key and the signature key are saved in the second authorized terminal.

10. A readable storage medium, characterized by, Comprising, a memory, the memory is configured to store instructions that, when executed by a processor, cause an apparatus including the readable storage medium to perform the method of any one of claims 1-5.

Images