A method and system for dynamic desensitization of large data unstructured files

Abstract

The invention discloses a big data non-structured file dynamic desensitization method and system. The system comprises a client, a central scheduler, a desensitization system and an HDFS, wherein the client sends a data reading instruction to the central scheduler and obtains data from a host where the desensitization system is located or the HDFS according to a DataNode address returned from the central scheduler; the central scheduler analyzes the data reading instruction and judges whether desensitization needs to be performed or not; if the desensitization needs to be performed, the central scheduler modifies the DataNode address of an obtained file data block to an address of the host where the desensitization system is located, returns the modified DataNode address to the client, and sends the unmodified DataNode address to the desensitization system; and the desensitization system is used for desensitizing data obtained from the DataNode address and returning the desensitized data to the client. Through the scheme, the realization is simple; the cost is reduced; the deployment is transparent; and existing applications are not influenced.

Description

technical field [0001] The invention relates to the field of data security, in particular to a method and system for dynamic desensitization of large data unstructured files. Background technique [0002] In the Hadoop ecosystem, the HDFS system is used to store unstructured data. While the big data processing system mines value from massive data, some archived data such as data dictionaries and population information serve as an important basis for data value mining. Once stored, they are rarely changed. HDFS files are a common storage method for this type of data. [0003] Using permission-based access control to make two authentication conclusions of yes or no for users' access to data resources, this method is difficult to meet the diverse data usage needs of users. When users who access certain types of sensitive data only need part of the sensitive data and do not need to obtain data with a higher security level, the permission control model alone cannot meet this dem...

AI Technical Summary

Problems solved by technology

[0006] (1) Once the desensitization is completed, the desensitization data is fixed. If other desensitization rules need to be applied, the desensitization process must be performed again

[0007] (2) An additional repository is required to store desensitized data

[0008] (3) Dynamic desensitization rule settings are not supported, and changes in desensitization rules cannot be reflected in desensitization data in real time

Images