Big data security-oriented multi-dimensional privacy protection and desensitization system

Abstract

The invention discloses a big data security-oriented multi-dimensional privacy protection and desensitization system, which comprises seven layers of architectures working cooperatively in sequence, and the seven layers of architectures comprise: a data access layer, which is used for receiving multi-source heterogeneous data and executing access security verification; the data preprocessing layer is used for cleaning and classifying the data output by the data access layer and identifying and grading sensitive information; the core desensitization layer is used for executing static desensitization by adopting a multi-dimensional privacy protection model based on the sensitive grading result output by the data preprocessing layer; according to the method, multi-dimensional protection can be carried out; a static desensitization k-anonymity model and dynamic calculation protection are combined, so that the full life cycle of data is covered; multi-source heterogeneous data access and identification are supported, and the sensitive information identification precision is greater than or equal to 95%; balancing privacy and availability: through hierarchical desensitization and effect verification, ensuring privacy compliance and retaining 90% or more of data analysis value at the same time; the whole process is controllable, operation traceability is achieved, and abnormity can be detected.

Description

Technical Field

[0001] This invention relates to the field of big data security and privacy protection technology, specifically a multi-dimensional privacy protection and de-identification system for big data security. Background Technology

[0002] To address the security and privacy challenges of the big data environment, existing technologies have shifted from traditional "perimeter protection" to a "data-centric" defense-in-depth system. Key technologies include: Privacy protection technologies, such as differential privacy and federated learning, provide strict mathematical privacy guarantees during data analysis and utilization, achieving "data usable but not visible." Data encryption and computational security: Homomorphic encryption allows direct computation on ciphertext; trusted execution environments provide hardware-level security isolation for data computation processes. Data usage control: Technologies such as data anonymization, access control, and data watermarking prevent data misuse or leakage during sharing, use, and circulation. Security governance and auditing: Through data classification and grading, data traceability, and security analysis, comprehensive management of the data lifecycle and proactive discovery of potential threats are achieved.

[0003] With the popularization of big data technology, the contradiction between data value and privacy security is becoming increasingly prominent. Existing technologies have the following shortcomings: limitations of single-dimensional protection: most systems only use static desensitization, lacking privacy protection in dynamic computing; poor adaptability: it is difficult to handle multi-source heterogeneous data, and the accuracy of sensitive information identification is low; imbalance between privacy and usability: excessive desensitization causes data to lose its analytical value, or insufficient protection leads to privacy leaks; weak controllability throughout the process: there is a lack of end-to-end monitoring and auditing of data from access to application, making it impossible to trace abnormal operations. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-dimensional privacy protection and de-identification system for big data security. It aims to adopt a seven-layer progressive architecture, with each layer working together to achieve full-process privacy protection for data access, preprocessing, de-identification, computation, storage, application and monitoring.

[0005] This invention is implemented as follows: a multi-dimensional privacy protection and de-identification system for big data security, comprising a seven-layer architecture that works sequentially and collaboratively, wherein the seven-layer architecture includes: The data access layer is used to receive heterogeneous data from multiple sources and perform access security verification. The data preprocessing layer is used to clean, classify, and identify and classify sensitive information in the data output by the data access layer. The core desensitization layer performs static desensitization based on the sensitivity classification results output by the data preprocessing layer using a multi-dimensional privacy protection model; The privacy computation layer performs dynamic privacy computation on anonymized or raw data, making the data usable but invisible. The secure storage layer performs encrypted storage and access control on the de-identified data output by the core de-identification layer and the calculation results output by the privacy computing layer; The application service layer encapsulates securely stored, anonymized data or calculation results into standardized service interfaces, which are then provided to upper-layer business applications for invocation. The monitoring and auditing layer performs log collection, anomaly detection, and audit tracing for the entire operation of the seven-layer architecture.

[0006] Preferably, the data access layer includes a multi-source adaptation module, an access security verification module, and a traffic control module; the multi-source adaptation module supports the access of batch static data, real-time streaming data, and external interface data; the access security verification module performs identity verification, data validity verification, and virus scanning; and the traffic control module limits the access rate based on the token bucket algorithm.

[0007] Preferably, the sensitive information identification and classification of the data preprocessing layer includes: performing regular expression matching and format matching on structured data through a rule engine, and performing sensitive entity identification on unstructured data through a pre-trained BERT model or YOLO+OCR; after identification, it is divided into five levels from L1 to L5 according to sensitivity, where L5 is the highest sensitivity level, corresponding to ID card number and bank card number data.

[0008] Preferably, the multidimensional privacy protection model of the core desensitization layer is a combination model of k-anonymity, l-diversity and t-proximity; the k-anonymity ensures that the number of individuals in each quasi-identifier equivalence class is ≥k, the l-diversity ensures that the different values ​​of sensitive attributes within each equivalence class are ≥l, and the t-proximity ensures that the difference in the distribution of sensitive attributes between the equivalence class and the whole set is ≤t.

[0009] Preferably, the privacy computing layer includes a federated learning submodule, a differential privacy submodule, and a homomorphic encryption submodule; the federated learning submodule supports horizontal or vertical federated training and achieves joint modeling through encrypted gradient parameter exchange; the differential privacy submodule adds noise through a Laplacian or exponential mechanism; and the homomorphic encryption submodule implements addition operations in the encrypted state based on the Paillier algorithm.

[0010] Preferably, the secure storage layer includes a storage encryption module, a key management module, and an access control module; the storage encryption module uses a TEE trusted execution environment combined with the AES-XTS algorithm to encrypt data; the key management module uses KMS to generate, store, distribute, rotate, and destroy keys, with the master key stored in the HSM hardware security module; and the access control module performs permission verification based on the RBAC+ABAC model.

[0011] Preferably, the application service layer includes an API gateway module, a service encapsulation module, and a service monitoring and fault tolerance module; the API gateway module performs routing management, identity authentication, and traffic limiting; the service encapsulation module provides data query, privacy calculation, and report generation services; and the service monitoring and fault tolerance module implements metric monitoring, circuit breaking, and degradation based on Prometheus and Resilience4j.

[0012] Preferably, the monitoring and auditing layer includes a full-process log collection module, an anomaly detection module, and an audit tracing module; the full-process log collection module collects operation logs at each level through Filebeat and Kafka and stores them in Elasticsearch; the anomaly detection module combines a rule engine and an isolated forest model to identify high-frequency access, abnormal permissions, and other operations; the audit tracing module supports full lifecycle operation backtracking of data and records log hash values ​​through blockchain to ensure immutability.

[0013] Preferably, the core desensitization layer further includes a desensitization effect verification module, used to verify whether the equivalence class size of the desensitized data satisfies k-anonymity, whether the entropy value of sensitive attributes satisfies l-diversity, whether the EMD distance satisfies t-proximity, and whether it supports core business analysis.

[0014] Preferably, the Laplace mechanism of the differential privacy submodule satisfies: for the query function f(D), the output result f*(D) = f(D) + Lap(Δf / ε), where Δf is the global sensitivity, ε is the privacy budget, and Lap(・) is the Laplace distribution noise.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention provides multi-dimensional protection: combining static desensitization k-anonymity model with dynamic computational protection to cover the entire data lifecycle; it also has high adaptability: supporting access and identification of multi-source heterogeneous data, with sensitive information identification accuracy ≥95%; it balances privacy and usability: through hierarchical desensitization and effect verification, it ensures privacy compliance while retaining ≥90% of the data analysis value; and it is fully controllable: enabling traceable operations and detectable anomalies. Attached Figure Description

[0016] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a flowchart of the process of the present invention; Detailed Implementation

[0017] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0018] The following description, in conjunction with the accompanying drawings and specific embodiments, provides further details: Example 1 like Figure 1 As shown, this invention discloses a multi-dimensional privacy protection and de-identification system for big data security. It adopts a seven-layer progressive architecture, with each layer working together to achieve end-to-end privacy protection for data access, preprocessing, de-identification, computation, storage, application, and monitoring. Specifically, it includes: Data access layer: includes a multi-source adaptation module, an access security verification module, and a traffic control module.

[0019] The multi-source adaptation module supports the access of batch static data HDFS, MySQL, Hive, real-time streaming data Kafka, Flume, and external interface data RESTfulAPI and FTP, and achieves format adaptation and synchronization through tools such as Sqoop and FlinkCDC; The access security verification module performs identity verification via LDAP / OAuth2.0, performs data validity verification based on JSONSchema, and calls the ClamAV engine to detect and remove viruses; The flow control module limits the access rate of a single data source based on the token bucket algorithm, and uses a Redis queue for buffering in high-concurrency scenarios.

[0020] Data preprocessing layer: includes data cleaning module, data classification module, and sensitive information identification and classification module.

[0021] The data cleaning module uses statistical methods to detect missing, duplicate, and outlier values, and identifies numerical anomalies using the IQR method. It then performs filling, deduplication, or discarding based on the data type, including numeric, character, and date types. The data classification module labels data according to business type and format, including structured / semi-structured / unstructured data; The sensitive information identification and classification module combines a rule engine with regular expressions to match structured data such as ID card numbers and mobile phone numbers, and the AI ​​model BERT to identify sensitive entities in unstructured text. YOLO+OCR is used to extract sensitive information from images, and the data is classified into levels L1-L5 according to sensitivity.

[0022] Core desensitization layer: Based on the sensitivity classification results, the corresponding desensitization algorithm is called through the algorithm scheduling engine, and static desensitization is performed by combining a multi-dimensional privacy protection model of k-anonymity, l-diversity, and t-proximity. For L5 level data, AES-256 encryption and desensitization are used, with salt value and CBC mode. For L4 level data, replacement / generalization desensitization is used, for L3 level data, range generalization is used, and for L2 level data, partial suppression is used. k-Anonymity ensures that the number of individuals in the equivalence class of each quasi-identifier is ≥k (k≥5), l-Diversity ensures that the value of the sensitive attribute is ≥l (l≥2) through entropy calculation, and t-Proximity controls the difference between the equivalence class and the distribution of the whole set to be ≤t (t≤0.3) through EMD distance. The desensitization effect verification module automatically verifies the privacy model's compliance and business availability.

[0023] Privacy-preserving computation layer: Includes federated learning submodule, differential privacy submodule, and homomorphic encryption submodule, to achieve dynamic computation privacy protection. The federated learning submodule is based on the FedAvg algorithm and achieves joint modeling of multiple institutions through encrypted gradient parameter exchange, with local data remaining within the domain. The differential privacy submodule adds noise through the Laplace mechanism f*(D)=f(D)+Lap(Δf / ε) or the exponential mechanism to protect the statistical query results; The homomorphic encryption submodule is based on the Paillier algorithm and supports addition operations in the encrypted state.

[0024] Secure storage layer: includes storage encryption module, key management module, and access control module. The storage encryption module uses IntelSGX / ARMTrustZone to build a TEE trusted execution environment, and combines the AES-XTS algorithm to encrypt data blocks (4KB / block), which are decrypted only within the TEE; The Key Management Module (KMS) manages keys through Shamir fragmentation, and the master key is stored in the HSM hardware security module, supporting automatic rotation (every 90 days). The access control module is based on the RBAC+ABAC model, which combines roles and attributes to restrict data access.

[0025] Application service layer: includes API gateway module, service encapsulation module, and service monitoring and fault tolerance module. The API gateway module implements route management, OAuth2.0 authentication, and Redis rate limiting through Spring Cloud Gateway; The service encapsulation module provides data query, privacy computation (federated training / differential query), and report generation services; The service monitoring and fault tolerance module is based on Prometheus+Grafana monitoring metrics (response time ≤500ms) and implements circuit breaking and degradation through Resilience4j.

[0026] Monitoring and auditing layer: includes a full-process log collection module, an anomaly detection module, and an audit tracing module. The log collection module uses Filebeat and Kafka to collect logs at various levels and stores them in Elasticsearch. The anomaly detection module combines the Drools rule engine with the isolated forest model; The audit traceability module supports full lifecycle data operation traceability. By inputting the data ID, the entire chain from access to de-identification to application can be traced. The log hash value is recorded through the FISCOBCOS blockchain to ensure immutability.

[0027] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and the medical big data scenario: Data access: Patient data from the hospital's HIS system is incrementally synchronized to the system via Oracle database through Sqoop. During access, API keys and IP whitelists are verified. During peak sales periods, the access rate is controlled at 1000 records / second through a Redis queue.

[0028] Data preprocessing: Cleaning: Fill in missing values ​​for "visit time" and delete abnormal data such as "age = 150 years old"; Category: Marked as "Patient Basic Information" and "PDF Medical Records"; Sensitive identification: Regular expression matching of ID card number (L5), BERT model to identify "lung cancer" in medical record (L4), YOLO+OCR to extract "name" in examination report (L3).

[0029] Core Desensitization: L5 level ID number: AES-256 encrypted, salt value "salt_2024", CBC mode; L4 grade "lung cancer": generalized to "malignant respiratory diseases"; Level L3 “35 years old”: generalized to “30-40 years old”; Verification: "30-40 years old + respiratory diseases" equivalence class size ≥ 5 (k=5), sensitive attribute entropy value ≥ 1.58 (l=3).

[0030] Privacy-preserving computation: Cancer prediction models are trained in collaboration with other hospitals, and encrypted gradients are exchanged through federated learning and Paillier encryption, ensuring that local data is not transmitted out. When calculating the "percentage of patients aged 30-40", Laplace noise (ε=0.5, Δf=0.001) is added.

[0031] Secure storage: De-identified data is stored in HBase after being encrypted with SGX. The AES key is managed by KMS, and the data is divided into 3 shards. It requires 2 people to decrypt and can only be accessed by the "attending physician" role during working hours.

[0032] Application service: The research team queries anonymized data through the " / api / medical / query" interface. The API gateway verifies the token and limits the flow. If the response times out, the cached result is returned.

[0033] Monitoring and Auditing: Real-time collection of API call logs; triggering WeChat alarms if "unauthorized IP access in the early morning" is detected; when tracing patient data, blockchain verification logs are confirmed to be tamper-proof, providing a complete view of the data flow chain. The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-dimensional privacy protection and de-identification system for big data security, characterized in that, This includes a seven-layer architecture that works sequentially and collaboratively, the seven-layer architecture comprising: The data access layer is used to receive heterogeneous data from multiple sources and perform access security verification. The data preprocessing layer is used to clean, classify, and identify and classify sensitive information in the data output by the data access layer. The core desensitization layer performs static desensitization based on the sensitivity classification results output by the data preprocessing layer using a multi-dimensional privacy protection model; The privacy computation layer performs dynamic privacy computation on anonymized or raw data, making the data usable but invisible. The secure storage layer performs encrypted storage and access control on the de-identified data output by the core de-identification layer and the calculation results output by the privacy computing layer; The application service layer encapsulates securely stored, anonymized data or calculation results into standardized service interfaces, which are then provided to upper-layer business applications for invocation. The monitoring and auditing layer performs log collection, anomaly detection, and audit tracing for the entire operation of the seven-layer architecture.

2. The multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The data access layer includes a multi-source adaptation module, an access security verification module, and a traffic control module. The multi-source adaptation module supports the access of batch static data, real-time streaming data, and external interface data. The access security verification module performs identity verification, data legality verification, and virus scanning. The traffic control module limits the access rate based on the token bucket algorithm.

3. The multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The sensitive information identification and classification of the data preprocessing layer includes: performing regular expression matching and format matching on structured data through a rule engine, and performing sensitive entity identification on unstructured data through a pre-trained BERT model or YOLO+OCR; after identification, it is divided into five levels from L1 to L5 according to sensitivity, with L5 being the highest sensitivity level, corresponding to ID card number and bank card number data.

4. A multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The multidimensional privacy protection model of the core desensitization layer is a combination of k-anonymity, l-diversity and t-proximity; k-anonymity ensures that the number of individuals in each quasi-identifier equivalence class is ≥k, l-diversity ensures that the different values ​​of sensitive attributes within each equivalence class are ≥l, and t-proximity ensures that the difference in the distribution of sensitive attributes between the equivalence class and the whole set is ≤t.

5. A multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The privacy computing layer includes a federated learning submodule, a differential privacy submodule, and a homomorphic encryption submodule. The federated learning submodule supports horizontal or vertical federated training and achieves joint modeling through encrypted gradient parameter exchange. The differential privacy submodule adds noise through a Laplacian or exponential mechanism. The homomorphic encryption submodule implements addition operations in the encrypted state based on the Paillier algorithm.

6. A multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The secure storage layer includes a storage encryption module, a key management module, and an access control module. The storage encryption module uses a Trusted Execution Environment (TEE) combined with the AES-XTS algorithm to encrypt data. The key management module uses KMS to generate, store, distribute, rotate, and destroy keys, with the master key stored in the HSM hardware security module. The access control module performs permission verification based on the RBAC+ABAC model.

7. A multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The application service layer includes an API gateway module, a service encapsulation module, and a service monitoring and fault tolerance module. The API gateway module performs routing management, identity authentication, and traffic limiting; the service encapsulation module provides data query, privacy calculation, and report generation services; and the service monitoring and fault tolerance module implements metric monitoring, circuit breaking, and degradation based on Prometheus and Resilience4j.

8. A multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The monitoring and auditing layer includes a full-process log collection module, an anomaly detection module, and an audit tracing module. The full-process log collection module collects operation logs at each level through Filebeat and Kafka and stores them in Elasticsearch. The anomaly detection module combines a rule engine and an isolated forest model to identify high-frequency access, abnormal permissions, and other operations. The audit tracing module supports full lifecycle operation backtracking of data and records log hash values ​​through blockchain to ensure immutability.

9. A multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The core desensitization layer also includes a desensitization effect verification module, which is used to verify whether the equivalence class size of the desensitized data satisfies k-anonymity, whether the entropy value of sensitive attributes satisfies l-diversity, whether the EMD distance satisfies t-proximity, and whether it supports core business analysis.

10. A multi-dimensional privacy protection and de-identification system for big data security according to claim 1, characterized in that, The Laplace mechanism of the differential privacy submodule satisfies: for the query function f(D), the output result f*(D)=f(D)+Lap(Δf / ε), where Δf is the global sensitivity, ε is the privacy budget, and Lap(・) is the Laplace distribution noise.

Images