Big Data Desensitization Methods

Abstract

The present invention discloses a big data desensitization method, which desensitizes the specified data in the multidimensional fact table, including: an initialization step, reading the specified data in the multidimensional fact table and arranging them into a data matrix, each column in the data matrix corresponds to a dimension, and the data matrix is ​​the original data matrix. A spatial transformation step, transforming the specified data of each dimension according to the column, the transformation includes a stretching transformation, a shrinking transformation or a twisting transformation, and obtaining a transformed data matrix. Among them, after normalization, the difference between the value of each data in the transformed data matrix and the corresponding value in the original data matrix is ​​less than 5%. The big data desensitization method of the present invention uses spatial transformation to desensitize sensitive data, and the spatial relative position information of the desensitized data is retained, and the data loss caused by spatial transformation is less than 5%. The big data desensitization method can also be applied to a distributed framework to meet the needs of distributed system big data computing.

Description

Technical Field

[0001] The present invention relates to the field of big data, and more specifically, to data security technology for big data. Background Art

[0002] Data processing is becoming an important infrastructure. For data processing, data security, especially the security of sensitive data, is particularly important. Data desensitization for sensitive data is also a kind of infrastructure. In the financial field, the prior art basically uses random value replacement desensitization and special character replacement desensitization for data desensitization. The former uses random value replacement (letters become random letters, numbers become random numbers) to change data, and the latter uses special characters (such as "*") to replace to change data.

[0003] This desensitization method is suitable for data that has no specific meaning and only serves as an indicator, such as name, mobile phone number, card number, etc. Indicative information such as name, mobile phone number, card number, etc. has no substantial meaning for data mining and data analysis.

[0004] With the continuous deepening of informatization and dataization in the financial industry, data mining and data analysis of financial data are becoming more and more important. Data mining and data analysis play an increasingly important role in risk control, risk warning, customer identification and efficiency improvement. In the process of data mining and data analysis, asset data, behavioral data, customer portraits and other data are needed. These data also belong to customer privacy and must be desensitized before use. For these data, the data itself has meaning. According to the traditional random value replacement desensitization method or special character replacement desensitization method, the data itself will be changed, and the meaning of the data itself will be partially or completely lost, which will make subsequent data mining and data analysis impossible. In addition, in order to obtain more efficient risk control, risk warning, customer identification and efficiency improvement, it is hoped that data can be shared among more financial institutions, and the big data owned by multiple financial institutions can be analyzed, which will be more accurate. Data sharing and exchange have higher requirements for data desensitization. On the one hand, it is required that the desensitized data is significantly different from the original data, and it is necessary to ensure that the original data cannot be restored or located based on the desensitized data to avoid attacks on the original data. On the other hand, the desensitized data should retain the meaning and information of the original data as much as possible, so that subsequent data mining and data analysis can continue and maintain the required accuracy. Summary of the invention

[0005] The present invention aims to propose a big data desensitization method, which can perform minimally desensitizing on sensitive data, and the method can also be executed on a distributed framework.

[0006] According to an embodiment of the present invention, a big data desensitization method is proposed to desensitize specified data in a multi-dimensional fact table, including:

[0007] Initialization step, reading the specified data in the multi-dimensional fact table and arranging it into a data matrix, each column in the data matrix corresponds to a dimension, and the data matrix is ​​the original data matrix;

[0008] A spatial transformation step, transforming the specified data of each dimension according to columns, wherein the transformation includes stretching transformation, shrinking transformation or twisting transformation to obtain a transformed data matrix;

[0009] Among them, after normalization processing, the difference between the value of each data in the transformed data matrix and the corresponding value in the original data matrix is ​​less than 5%.

[0010] In one embodiment, the stretching transformation includes: generating a column of random magnification coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random magnification coefficients satisfies a normal distribution; multiplying the column of specified data with the column of random magnification coefficients to obtain a column that has been stretched. The shrinking transformation includes: generating a column of random shrinking coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random shrinking coefficients satisfies a normal distribution; multiplying the column of specified data with the column of random shrinking coefficients to obtain a column that has been shrunk.

[0011] In one embodiment, the distortion transformation includes: generating a Sigmod function; generating a column of random additional coefficients, the number of the random additional coefficients is the same as the number of specified data in the corresponding column, and the column of random additional coefficients satisfies the normal distribution; using the Sigmod function to operate on the column of specified data, and the result of the operation and the corresponding random additional coefficients together obtain a column that has been distorted.

[0012] In one embodiment, a transformation matrix is ​​generated according to the transformation of the specified data of each dimension, each column in the transformation matrix corresponds to a dimension of the specified data, and the transformation matrix is ​​multiplied with the original data matrix to obtain a transformed data matrix.

[0013] In one embodiment, the method further includes: encrypting the remaining data except the specified data in the multi-dimensional fact table.

[0014] According to an embodiment of the present invention, a distributed big data desensitization method is proposed, which desensitizes specified data in a multi-dimensional fact table in a distributed framework, including:

[0015] In the mapping step, the mapper reads the data in the multi-dimensional fact table and arranges it into a data matrix. Each column in the data matrix corresponds to a dimension. Each dimension uses an independent mapper. The data matrix is ​​an unfiltered data matrix.

[0016] The screening step is to screen the unscreened data matrix, select the column where the specified data is located to form the original data matrix, and the remaining data except the specified data forms the auxiliary data matrix;

[0017] a spatial transformation step, transforming the designated data of each dimension according to the columns, wherein the transformation includes stretching transformation, shrinking transformation or twisting transformation, and the transformation of each dimension is processed by an independent reducer, and a transformed data matrix is ​​obtained after reduction, wherein after normalization processing, the difference between the value of each data in the transformed data matrix and the corresponding value in the original data matrix is ​​less than 5%;

[0018] In the data merging step, the data in the auxiliary data matrix is ​​encrypted, and the encrypted auxiliary data matrix is ​​merged with the transformed data matrix to generate a desensitized data matrix.

[0019] In one embodiment, the stretching transformation includes: generating a column of random magnification coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random magnification coefficients satisfies a normal distribution; multiplying the column of specified data with the column of random magnification coefficients to obtain a column that has been stretched. The shrinking transformation includes: generating a column of random shrinking coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random shrinking coefficients satisfies a normal distribution; multiplying the column of specified data with the column of random shrinking coefficients to obtain a column that has been shrunk.

[0020] In one embodiment, the distortion transformation includes: generating a Sigmod function; generating a column of random additional coefficients, the number of the random additional coefficients is the same as the number of specified data in the corresponding column, and the column of random additional coefficients satisfies the normal distribution; using the Sigmod function to operate on the column of specified data, and the result of the operation and the corresponding random additional coefficients together obtain a column that has been distorted.

[0021] In one embodiment, the reducer generates a column of transformation matrices according to the transformation of the specified data of each dimension, and the reducer multiplies the column of transformation matrices with a corresponding column of specified data in the original data matrix to obtain a corresponding column of data in the transformed data matrix.

[0022] The big data desensitization method of the present invention uses spatial transformation to desensitize sensitive data, and the spatial relative position information of the desensitized data is retained. From the perspective of data mining and data analysis, the data loss caused by spatial transformation is less than 5%, so the method does not affect subsequent data processing. The big data desensitization method can also be applied to a distributed framework to meet the needs of distributed system big data computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a big data desensitization method according to an embodiment of the present invention is disclosed.

[0024] Figure 2 A flow chart of a big data desensitization method according to another embodiment of the present invention is disclosed. DETAILED DESCRIPTION

[0025] The present invention proposes a big data desensitization method to desensitize specified data in a multi-dimensional fact table. Figure 1 A flowchart of a big data desensitization method according to an embodiment of the present invention is disclosed. As shown in the figure, the method includes:

[0026] S101, initialization step, reading the specified data in the multi-dimensional fact table and arranging it into a data matrix, each column in the data matrix corresponds to a dimension, and the data matrix is ​​the original data matrix.

[0027] In the financial industry, customer data is usually stored in fact tables. In the fact table, some data is used for customer identification, such as name, card number, address, mobile phone number, etc.; some data records customer asset information, such as total assets, RMB assets, foreign currency assets, financial products, credit cards, deposits, bonds, etc.; some data records customer behavior information, such as transfer records, purchase records, commodity transaction records, etc. Among them, the identification data is basically not used for subsequent data analysis, so it can be encrypted or desensitized using traditional methods. Even if the information contained in the identification data is lost or lost during the encryption or desensitization process, it will basically not affect the results of data analysis. Asset information data and behavior information data are the core data for data analysis and data mining. After desensitization, the meaning and information of the original data need to be retained to ensure the effectiveness and accuracy of data analysis. In the present invention, data such as asset information data and behavior information data are called designated data, and the designated data in the fact table will be desensitized using the big data desensitization method proposed by the present invention.

[0028] Table 1

[0029] card number Total financial management Credit card bill Total national debt 161340511092313455 195223.24 1642.09 50000.00 161421686113451004 20000.00 10821.45 20000.00 161731799840912249 85000.24 411.16 50000.00

[0030] Table 1 shows an example of a fact table that records customer data. Table 1 is in the form of a two-dimensional table, which is the most widely used way to record customer data. In the two-dimensional table shown in Table 1, each row represents the data of a customer, and each column represents a category. In the present invention, each "category" represents a dimension, and the "dimension" of the present invention represents the data of all customers under a certain category. The dimension "card number" in Table 1 is identity identification data, and "total financial management amount", "credit card bill" and "total national debt amount" are asset information data. Behavioral information data is not listed in Table 1. In Table 1, the data of the three dimensions of "total financial management amount", "credit card bill" and "total national debt amount" are designated data.

[0031] Table 2

[0032] 195223.24 1642.09 50000.00 20000.00 10821.45 20000.00 85000.24 411.16 50000.00

[0033] In step S101, the specified data in the multi-dimensional fact table (Table 1) is read and arranged into a data matrix (Table 2), each column in the data matrix corresponds to a dimension, and the data matrix (Table 2) is the original data matrix.

[0034] S102, a spatial transformation step, transforming the designated data of each dimension according to the columns, wherein the transformation includes stretching transformation, shrinking transformation or twisting transformation, to obtain a transformed data matrix. After normalization, the difference between the value of each data in the transformed data matrix and the corresponding value in the original data matrix is ​​less than 5%.

[0035] In one embodiment,

[0036] The stretch transformations include:

[0037] Generate a column of random magnification coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random magnification coefficients satisfies a normal distribution;

[0038] The specified data of the column is multiplied by the random magnification coefficient of the column to obtain a stretched column.

[0039] The shrinking transformations include:

[0040] Generate a column of random shrinkage coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random shrinkage coefficients satisfies the normal distribution;

[0041] The specified data of the column is multiplied by the random shrinkage coefficient of the column to obtain a column that has been shrunken.

[0042] The distortion transformations include:

[0043] Generate a Sigmod function;

[0044] Generate a column of random additional coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random additional coefficients satisfies the normal distribution;

[0045] The Sigmod function is used to operate the specified data of the column, and the result of the operation and the corresponding random additional coefficient are used together to obtain the distorted transformed column.

[0046] The main method of data desensitization for the specified data in the present invention is spatial transformation. The original value is hidden and replaced with a new value through stretching transformation, shrinking transformation or twisting transformation, but the relative spatial position relationship between each new value remains unchanged or changes very little, so that the results of subsequent data mining and data analysis are basically unaffected, and the results of data analysis on the desensitized data are basically consistent with the results of data analysis on the original data, or in other words, the difference is controlled within 5%.

[0047] For example, the original data matrix shown in Table 2 is stretched and transformed. First, a column of random magnification coefficients is generated, the number of which is the same as the number of specified data in the corresponding column, and the column of random magnification coefficients satisfies the normal distribution. Then, the column of specified data is multiplied by the column of random magnification coefficients to obtain a stretched and transformed column.

[0048] The implementation code of an example of stretch transformation is as follows:

[0049]

[0050] The to_random() function generates a column of random magnification factors, and the number of random magnification factors is the same as the number of specified data in the corresponding column. The characteristic of the to_random() function is that the generated random numbers conform to the normal distribution, so the column of random magnification factors generated by to_random() satisfies the normal distribution.

[0051] Fin refers to the dimension "total financial management amount", credit refers to the dimension "credit card bill", and bond refers to the dimension "total government debt".

[0052] Multiply a column of specified data by the random magnification coefficient corresponding to the column to obtain a stretched column. After multiplication, the following data in Table 3 are obtained:

[0053] Table 3

[0054] 710264.3341476232 6455.808978154993 258916.91251466938 72764.32192679757 42544.083495213636 103566.76500586775 309249.2413607528 1616.458549445041 258916.91251466938

[0055] The data matrix in Table 3 is the transformed data matrix.

[0056] Verify the relative spatial position relationship of each data in the original data matrix and the transformed data matrix respectively. The verification method is to normalize the data matrix. Then calculate the Euclidean distance between each data point in the data matrix. The Euclidean distance is calculated by the function euclidean_distance(coords1,coords2).

[0057] Normalization uses the MinMaxScaler() function and executes the following sample code on the original data matrix:

[0058] min_max_scaler=preprocessing.MinMaxScaler()

[0059] result_origin=min_max_scaler.fit_transform(np.array(origin))

[0060] The normalized verification data matrix table 4 is obtained:

[0061] Table 4

[0062] 1 0.11824166 1 0 1 0 0.37095673 0 1

[0063] The following example code is executed on the transformed data matrix:

[0064] min_max_scaler=preprocessing.MinMaxScaler()

[0065] result_transformed=min_max_scaler.fit_transform(np.array(transformed))

[0066] The normalized verification data matrix table 5 is obtained:

[0067] Table 5

[0068] 1 0.11824166 1 0 1 0 0.37095673 0 1

[0069] Table 4 and Table 5 are consistent at the level of 8 decimal places, indicating that the relative spatial position relationship of each data in the original data matrix and the transformed data matrix remains basically unchanged, that is, the Euclidean distance between each data point after normalization is the same, which can be considered to be completely consistent.

[0070] The calculation method of shrinking transformation is similar to that of stretching transformation, so it will not be described again here. Both stretching transformation and shrinking transformation are linear transformations, so the relative spatial position relationship of each data in the original data matrix and the transformed data matrix remains basically unchanged, and it can be considered as a lossless transformation under the accuracy required for data analysis.

[0071] The distortion transformation is further explained by taking the original data matrix shown in Table 2 as an example. The distortion transformation first generates a Sigmod function. A column of random additional coefficients is generated, and the number of random additional coefficients is the same as the number of specified data in the corresponding column, and the column of random additional coefficients satisfies the normal distribution. Then, the Sigmod function is used to operate the specified data in the column, and the result of the operation and the corresponding random additional coefficient are used together to obtain the column after distortion transformation.

[0072] In one embodiment, the Sigmod function uses sigmoid(x), an S-shaped growth curve function. The Sigmod function is defined using the following code:

[0073] def sigmoid(x):

[0074] return 1. / (1+np.exp(-x))

[0075] In one embodiment, the random additional coefficients are generated by random.normalvariate. The following code is used to generate the random additional coefficients:

[0076] def to_random_within(i):

[0077] return random.normalvariate(i,1)

[0078] The random additional coefficients generated by random.normalvariate satisfy the normal distribution.

[0079] The Sigmod function is used to operate the column of specified data, and the result of the operation and the corresponding random additional coefficient are used together to obtain the twisted column. The transformed data matrix after the twisted transformation is obtained through the following example code.

[0080]

[0081] The to_random_within function generates a column of random additional coefficients, and the number of random additional coefficients is the same as the number of specified data in the corresponding column. The characteristic of the to_random_within function is that the generated random numbers conform to the normal distribution, so the column of random additional coefficients generated by to_random_within satisfies the normal distribution.

[0082] Fin refers to the dimension "total financial management amount", credit refers to the dimension "credit card bill", and bond refers to the dimension "total government debt".

[0083] The Sigmod function is used to operate a column of specified data. The result of the operation is added to the corresponding random additional coefficient to obtain a distorted column. After calculating each column separately, a distorted data matrix is ​​obtained, as shown in Table 6:

[0084] Table 6

[0085] 2.9818167618925275 -0.7316915640653386 1.9027560487948123 2.7507581832625227 -0.5301590086516604 1.6716974701648073 2.8424483209688303 -0.7612175872816653 1.9027560487948123

[0086] The following sample code is executed to normalize the distorted data matrix (Table 6):

[0087] result_transformed=min_max_scaler.fit_transform(np.array(transformed))

[0088] print(result_transformed)

[0089] The normalized verification data matrix table 7 is obtained:

[0090] Table 7

[0091] 1 0.12778588 1 0 1 0 0.39682637 0 1

[0092] Although there are some differences in the values ​​between Table 4 and Table 7, the difference is less than 5%, indicating that there is only a slight change in the relative spatial position relationship between the original data matrix and the transformed data matrix. There is a slight change in the relative spatial position of each data in the original data matrix and the distorted data matrix, that is, there is a certain difference between the Euclidean distance between each data point in the normalized original data matrix and the Euclidean distance between each data point in the transformed data matrix, but this change difference is less than 5% after normalization, which is a slight loss transformation for the accuracy required for data analysis and will not affect the results of data analysis.

[0093] In one embodiment, when generating a transformed data matrix from an original data matrix, a transformation matrix is ​​generated according to the transformation of the specified data of each dimension, each column in the transformation matrix corresponds to a dimension of the specified data, and the transformation matrix is ​​multiplied with the original data matrix to obtain a transformed data matrix. For the data of each dimension (i.e., each column) in the original data matrix, the same transformation rule can be applied, or different transformation rules can be applied. According to the difference in the transformation rules, a column of multipliers is generated for the specified data of each column, and the column multiplier can be a Sigmod function of a random magnification coefficient, a random contraction coefficient, or a superimposed random additional coefficient. The multipliers of each column are combined into a transformation matrix, each column in the transformation matrix corresponds to a dimension of the specified data, and the transformation matrix is ​​multiplied with the original data matrix to obtain a transformed data matrix.

[0094] In one embodiment, the big data desensitization method of the present invention further includes: encrypting the remaining data in the multidimensional fact table except for the specified data. Continuing with the multidimensional fact table shown in Table 1 as an example, the three columns of "total financial management amount", "credit card bill" and "total national debt amount" are the specified data. The "card number" column is the remaining data. Therefore, the card number column is encrypted. In one embodiment, the card number is encrypted using the MD5 algorithm, which is executed by the following example code:

[0095] cust_ids=[to_md5(x[0])for x in sample_data]

[0096] Where cust_id represents the card number.

[0097] The encrypted data will cause data information loss, but the card number will not be involved in the subsequent data analysis, so the loss of data information will not affect the results of data analysis. In addition to the encryption algorithm, the rest of the data except the specified data can also be desensitized by replacing it with random values ​​or special characters.

[0098] As financial institutions increasingly use distributed architectures for big data storage, in order to adapt to the big data storage method of distributed architectures, the big data desensitization method of the present invention can also be implemented on distributed architectures. Figure 2 As shown, Figure 2A flow chart of a big data desensitization method according to another embodiment of the present invention is disclosed. This embodiment proposes a distributed big data desensitization method, which desensitizes specified data in a multidimensional fact table under a distributed framework. The method is based on a map / reduce distributed framework as a whole. It can be linearly expanded and is applicable to various Hadoop components, including but not limited to hbase, hive, impala, etc. The distributed big data desensitization method includes the following steps:

[0099] S201, mapping step: A mapper reads the data in the multi-dimensional fact table and arranges it into a data matrix, each column in the data matrix corresponds to a dimension, and each dimension uses an independent mapper. The data matrix is ​​an unscreened data matrix.

[0100] At present, the mainstream storage method of financial big data is Parquet format, which is a column storage format suitable for Hadoop. If the multidimensional fact table is stored in Parquet format, it is a column storage format itself, and can be directly read by column through the mapper. If the multidimensional fact table is not in column storage format, but in row storage format (such as TXT format), then a format conversion step will be included before the mapping step. The format conversion step is also based on the map / reduce distributed framework. In the format conversion step, a mapper is applied to each row of the multidimensional fact table in row storage format. The mapper reads a row of data and cuts it into multiple segments according to the dimension. For example, for an N-dimensional fact table, the mapper reads a row of data and cuts the row of data into N segments. If the N-dimensional fact table has M rows, M mappers are required to process it. After the mapping is completed, the reducer performs reduction. The reduction is performed according to the dimension, and a reducer is set for each dimension. For an N-dimensional fact table, N reducers are required. The data reduced by the reducer is stored in columns. In this way, after the mapping / reduction process of the format conversion step is completed, the N-dimensional fact table stored in rows is converted to an N-dimensional fact table stored in columns. Then the mapping step is continued.

[0101] S202, screening step. Screen the unscreened data matrix, select the column where the specified data is located to form the original data matrix, and the remaining data except the specified data forms the auxiliary data matrix. Take the fact table shown in Table 1 as an example for explanation. Table 1 is a fact table. After the mapping step, an unscreened data matrix as shown in Table 8 is formed:

[0102] Table 8

[0103] 161340511092313455 195223.24 1642.09 50000.00 161421686113451004 20000.00 10821.45 20000.00 161731799840912249 85000.24 411.16 50000.00

[0104] After screening, the unscreened data matrix is ​​divided into the original data matrix (refer to the aforementioned Table 2) and the auxiliary data matrix (refer to the following Table 9). The original data matrix includes the specified data: asset information data of three dimensions: "total amount of financial management", "credit card bills" and "total amount of national debt". The auxiliary data matrix includes identity recognition data, as shown in Table 9:

[0105] Table 9

[0106] 161340511092313455 161421686113451004 161731799840912249

[0107] In other embodiments, the following screening principles are applied in the screening step to distinguish between the designated data and the auxiliary data:

[0108] Determine the data type: whether it is continuous value data or discrete data.

[0109] Continuous value data is classified as specified data. If the continuous value data is time continuous data, the data format is converted to timestamp, and the minimum value is set as 0. The difference between the remaining time continuous data and the minimum value is retained and set to double type. If it is digital continuous data, the original data is retained.

[0110] For discrete data, determine whether the number of different values ​​contained in the field is greater than 65536.

[0111] For discrete data whose field contains no more than 65536 different values, convert them into one-hot encoding and classify them as specified data.

[0112] For discrete data with more than 65536 different values, it is classified as auxiliary data. The original field content is retained. In most cases, this is a unique identifier such as a card number, customer number, or internal log number.

[0113] S203, spatial transformation step. The designated data of each dimension is transformed according to the columns, and the transformation includes stretching transformation, shrinking transformation or twisting transformation. The transformation of each dimension is processed by an independent reducer, and a transformed data matrix is ​​obtained after reduction. After normalization, the difference between the value of each data in the transformed data matrix and the corresponding value in the original data matrix is ​​less than 5%.

[0114] In one embodiment,

[0115] The stretch transformations include:

[0116] Generate a column of random magnification coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random magnification coefficients satisfies a normal distribution;

[0117] The specified data of the column is multiplied by the random magnification coefficient of the column to obtain a stretched column.

[0118] The shrinking transformations include:

[0119] Generate a column of random shrinkage coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random shrinkage coefficients satisfies the normal distribution;

[0120] The specified data of the column is multiplied by the random shrinkage coefficient of the column to obtain a column that has been shrunken.

[0121] The distortion transformations include:

[0122] Generate a Sigmod function;

[0123] Generate a column of random additional coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random additional coefficients satisfies the normal distribution;

[0124] The Sigmod function is used to operate the specified data of the column, and the result of the operation and the corresponding random additional coefficient are used together to obtain the distorted transformed column.

[0125] The stretching transformation, contraction transformation and distortion transformation in step S203 are consistent with the operation rules of each transformation in the aforementioned step S102. The specific operation rules are not repeated here. The difference of step S203 is that when executed on a distributed framework, an independent reducer is used to process each dimensional transformation, and the transformed data matrix is ​​obtained after reduction. Each dimension uses its own independent reducer, so each dimension can have its own transformation method, and each dimension can use the same transformation or different transformations. In one embodiment, each reducer generates a column of transformation matrices based on the transformation of the specified data of each dimension. The reducer multiplies the column of transformation matrices with the corresponding column of specified data in the original data matrix to obtain the corresponding column of data in the transformed data matrix.

[0126] S204, data merging step. The data in the auxiliary data matrix is ​​encrypted, and the encrypted auxiliary data matrix and the transformed data matrix are merged to generate a desensitized data matrix. For the data in the auxiliary data matrix, since the auxiliary data does not participate in subsequent data mining and data analysis, it can be encrypted or desensitized in a lossy manner, such as the MD5 encryption algorithm introduced above, random value replacement or special character replacement in traditional desensitization, etc. The transformed data matrix stores the specified data after spatial transformation, and the specified data is desensitized by lossless or minimally damaged spatial transformation, but the information carried by the data is retained and can continue to be used for data analysis and data mining. The encrypted auxiliary data matrix and the transformed data matrix are merged to generate a desensitized data matrix. The desensitized data matrix can be saved as a file according to demand, such as a column storage file in Parquet format.

[0127] The big data desensitization method of the present invention uses spatial transformation to desensitize sensitive data, and the spatial relative position information of the desensitized data is retained. From the perspective of data mining and data analysis, the data loss caused by spatial transformation is less than 5%, so the method does not affect subsequent data processing. The big data desensitization method can also be applied to a distributed framework to meet the needs of distributed system big data computing.

[0128] It should also be noted that the embodiments listed above are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and similar changes or deformations made therewith can be directly derived or easily associated with the contents disclosed by those skilled in the art from the present invention, and should all belong to the protection scope of the present invention. The above embodiments are provided to those familiar with the art to implement or use the present invention, and those familiar with the art can make various modifications or changes to the above embodiments without departing from the inventive concept of the present invention. Therefore, the protection scope of the present invention is not limited by the above embodiments, but should be the maximum scope of the innovative features mentioned in the claims.

Claims

1. A big data desensitization method, characterized in that: Desensitizes specified data in multi-dimensional fact tables, including: Initialization step, reading the specified data in the multi-dimensional fact table and arranging it into a data matrix, each column in the data matrix corresponds to a dimension, and the data matrix is ​​the original data matrix; The spatial transformation step transforms the designated data of each dimension according to the columns, wherein the transformation includes stretching transformation, shrinking transformation or twisting transformation to obtain a transformed data matrix; wherein the stretching transformation includes: generating a column of random magnification coefficients, wherein the number of random magnification coefficients is the same as the number of designated data in the corresponding column, and the random magnification coefficients of the column satisfy a normal distribution; multiplying the designated data of the column by the random magnification coefficients of the column to obtain a stretched transformed column; the twisting transformation includes: generating a Sigmod function; generating a column of random additional coefficients, wherein the number of random additional coefficients is the same as the number of designated data in the corresponding column, and the random additional coefficients of the column satisfy a normal distribution; using the Sigmod function to operate on the designated data of the column, and adding the operation result to the corresponding random additional coefficient to obtain a twisted transformed column; After normalization, the difference between the value of each data in the transformed data matrix and the corresponding value in the original data matrix is ​​less than 5%.

2. The big data desensitization method according to claim 1, characterized in that: The shrinking transformation includes: Generate a column of random shrinkage coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random shrinkage coefficients satisfies the normal distribution; The specified data of the column is multiplied by the random shrinkage coefficient of the column to obtain a column that has been shrunken.

3. The big data desensitization method according to claim 1, characterized in that: A transformation matrix is ​​generated according to the transformation of the specified data of each dimension, each column in the transformation matrix corresponds to a dimension of the specified data, and the transformation matrix is ​​multiplied with the original data matrix to obtain a transformed data matrix.

4. The big data desensitization method according to claim 1, characterized in that: Also includes: Encrypts all data except the specified data in the multidimensional fact table.

5. A distributed big data desensitization method, characterized in that: Desensitizes specified data in multi-dimensional fact tables in a distributed framework, including: In the mapping step, the mapper reads the data in the multi-dimensional fact table and arranges it into a data matrix. Each column in the data matrix corresponds to a dimension. Each dimension uses an independent mapper. The data matrix is ​​an unfiltered data matrix. The screening step is to screen the unscreened data matrix, select the column where the specified data is located to form the original data matrix, and the remaining data except the specified data forms the auxiliary data matrix; The spatial transformation step transforms the designated data of each dimension according to the column, and the transformation includes stretching transformation, shrinking transformation or twisting transformation. The transformation of each dimension is processed by an independent reducer, and the transformed data matrix is ​​obtained after reduction, wherein the stretching transformation includes: generating a column of random magnification coefficients, the number of random magnification coefficients is the same as the number of designated data in the corresponding column, and the random magnification coefficients of the column satisfy the normal distribution; multiplying the designated data of the column by the random magnification coefficients of the column to obtain the stretched transformed column; the twisting transformation includes: generating a Sigmod function; generating a column of random additional coefficients, the number of random additional coefficients is the same as the number of designated data in the corresponding column, and the random additional coefficients of the column satisfy the normal distribution; using the Sigmod function to operate the designated data of the column, and adding the result of the operation to the corresponding random additional coefficient to obtain the twisted transformed column; wherein, after normalization, the difference between the value of each data in the transformed data matrix and the corresponding value in the original data matrix is ​​less than 5%; In the data merging step, the data in the auxiliary data matrix is ​​encrypted, and the encrypted auxiliary data matrix is ​​merged with the transformed data matrix to generate a desensitized data matrix.

6. The distributed big data desensitization method according to claim 5, characterized in that: The shrinking transformation includes: Generate a column of random shrinkage coefficients, the number of which is the same as the number of specified data in the corresponding column, and the column of random shrinkage coefficients satisfies the normal distribution; The specified data of the column is multiplied by the random shrinkage coefficient of the column to obtain a column that has been shrunken.

7. The distributed big data desensitization method according to claim 5, characterized in that: The reducer generates a column of transformation matrices based on the transformation of the specified data in each dimension, and multiplies the column of transformation matrices with a corresponding column of specified data in the original data matrix to obtain a column of data in the transformed data matrix corresponding to the second-instance claim modification replacement page 204341CNCN.

Images