Sensitive data management method based on big data analysis

Abstract

The invention discloses a sensitive data governance method based on big data analysis, which comprises the following steps: firstly, establishing a dynamic data governance framework including dynamic role and authority management, data consanguinity recording, automatic data discovery and classification and data security management; a hierarchical management data center is established, and standardized management of data is realized; and a self-adaptive data governance strategy: constructing a data governance strategy based on big data analysis for the dynamic data governance framework. According to the invention, intelligent role and authority management, comprehensive data consanguinity recording, automatic data discovery and classification and enhanced data security measures are realized, and a governance strategy can be continuously optimized according to a real-time data analysis result, so that the efficiency, accuracy and security of data governance are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data governance, and in particular relates to a sensitive data governance method based on big data analysis. Background Art

[0002] Enterprise data governance is the process of data management, data extraction, and data standardization for data published in enterprise information systems. Data governance is the prerequisite for ensuring the correct use of enterprise data. Through efficient data governance, enterprises can extract important information indicators related to scientific research and production from a large amount of process business data. It is the first step to trace product quality through big data. Data governance can also help effectively monitor product process data throughout the entire process of product development, technology, and production, so as to achieve the goal of predicting various product performance indicators in advance and promote the control of the entire product life cycle.

[0003] Currently, data governance has the following deficiencies:

[0004] 1) Traditional access control mechanisms are often static and difficult to automatically adjust based on user behavior and context. Even if there are adaptive mechanisms, they may lack sufficient intelligence to accurately evaluate user attributes, environmental factors, and business needs;

[0005] 2) Existing data governance may not be able to handle large-scale data sources or unstructured data well. At the same time, the automatic identification of sensitive data and key business data may not be accurate enough;

[0006] 3) Existing data governance strategies often lack sufficient flexibility to cope with the rapidly changing business environment. The strategy generation and optimization process may be manual and inefficient. Summary of the invention

[0007] The purpose of the present invention is to provide a sensitive data governance method based on big data analysis, which realizes intelligent role and authority management, comprehensive data lineage records, automated data discovery and classification, enhanced data security measures, and can continuously optimize governance strategies based on real-time data analysis results, thereby improving the efficiency, accuracy and security of data governance.

[0008] In order to achieve the purpose of the present invention, the technical solution adopted is:

[0009] A sensitive data governance method based on big data analysis includes the following steps:

[0010] S1. Establish a dynamic data governance framework, including dynamic role and permission management, data lineage records, automated data discovery and classification, and data security management;

[0011] S2. Establish a hierarchical management data center to achieve standardized data management;

[0012] S3, adaptive data governance strategy, builds a data governance strategy based on big data analysis for the dynamic data governance framework.

[0013] As an optimization solution of the present invention, dynamic role and permission management includes role definition and permission allocation. When defining roles, roles and permissions are automatically adjusted according to user behavior and context; when allocating permissions, an adaptive access control mechanism is adopted to dynamically adjust permissions according to user attributes, environmental factors and business needs.

[0014] As an optimization scheme of the present invention, data lineage records specifically include: using data integration tools to automatically scan various data sources within the enterprise, establishing connections with various data sources, and ensuring that metadata information can be read and written; extracting metadata information from various data sources, including table structure, field description, and data type, storing the extracted metadata in a central metadata warehouse, and building a data lineage map to display the source, conversion process, and usage of the data; using NLP and machine learning technology to automatically analyze the content of the data, identify sensitive data and key business data, and automatically generate data labels based on the analysis results; analyzing the dependencies between data sets to determine the impact of changes in a certain data set on other data sets; monitoring the flow of data between different systems in real time and recording detailed log information; using machine learning models to detect abnormal data flow behavior and issue early warnings in a timely manner.

[0015] As an optimization solution of the present invention, automated data discovery and classification uses NLP technology to parse text data, extract keywords, entities and semantic information, perform feature extraction and analysis on unstructured data such as images, audio and video, use machine learning algorithms to train data classification models, classify data according to data features, and automatically generate data labels based on the results of the classification model to facilitate subsequent data management and use; use regular expressions, pattern matching and machine learning algorithms to identify sensitive data, mark the identified sensitive data, and record its location and access status, define data quality rules and indicators, automatically evaluate data quality according to the defined rules, and generate a data quality report.

[0016] As an optimization solution of the present invention, data security management adds tags to sensitive data, encrypts sensitive data during data processing, anonymizes identity information, uses machine learning models to detect abnormal data access behaviors, and realizes instant data backup through LVM Snapshot technology.

[0017] As an optimization solution of the present invention, in step S2, it specifically includes: when establishing a hierarchically managed data center, classifying the data according to the frequency of use of the data and configuring the corresponding access speed.

[0018] As an optimization solution of the present invention, in step S3, it includes:

[0019] 31) Using machine learning algorithms to train models to predict key indicators of data governance, including data quality, data security, and data retrieval speed;

[0020] 32) Generate adaptive data governance strategies based on data analysis results. The data governance strategies include automated data cleaning rules, security vulnerability detection and repair strategies, data lifecycle management strategies, and access rights adjustment strategies;

[0021] 33) Apply adaptive data governance strategies to data governance, establish feedback mechanisms, collect feedback and logs, and continuously improve data governance strategies.

[0022] The present invention has positive effects:

[0023] 1. The present invention can significantly enhance the security of enterprise data through measures such as dynamic role and permission management, sensitive data identification and labeling, data encryption, and abnormal access behavior detection. This helps prevent unauthorized access, data leakage and other security threats, and ensures that the key information assets of the enterprise are properly protected;

[0024] 2. The application of automated data discovery and classification, data cleaning rules (such as missing value processing, outlier detection, duplicate record processing) and data quality assessment indicators of the present invention enables enterprises to continuously monitor and improve data quality. High-quality data is the basis for effective data analysis and decision support, which helps to improve business efficiency and accuracy;

[0025] 3. The present invention can greatly improve the data retrieval speed by managing data in layers (, configuring corresponding storage media and cache mechanisms. This not only improves the user experience, but also provides strong support for real-time analysis and processing;

[0026] 4. The present invention uses the data lineage record function to track the entire process of data from source to final use, helping enterprises better understand the source, conversion process and usage of data. This transparency is crucial to ensure data consistency and compliance;

[0027] 5. The present invention uses machine learning algorithms, natural language processing technology and automation tools (such as ETL job parser, data integration tool Apache NiFi, etc.) to automate many tasks that traditionally need to be completed manually. This can not only reduce human errors, but also significantly reduce operation and maintenance costs;

[0028] 6. The present invention establishes a feedback mechanism, regularly collects user feedback and system logs, and uses data analysis technology to continuously adjust and improve existing data governance strategies. This approach enables enterprises to respond quickly when faced with new challenges and maintain the effectiveness and relevance of their data governance system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0031] like Figure 1 As shown, a sensitive data governance method based on big data analysis includes the following steps:

[0032] S1. Establish a dynamic data governance framework, including dynamic role and permission management, data lineage records, automated data discovery and classification, and data security management;

[0033] S2. Establish a hierarchical management data center to achieve standardized data management;

[0034] S3, adaptive data governance strategy, builds a data governance strategy based on big data analysis for the dynamic data governance framework.

[0035] Among them: Dynamic role and permission management includes role definition and permission allocation. When defining roles, roles and permissions are automatically adjusted according to user behavior and context; when allocating permissions, an adaptive access control mechanism is used to dynamically adjust permissions according to user attributes, environmental factors and business needs. Record user operation logs, including login time, resources accessed and operations performed; use clustering algorithms to identify user behavior patterns; set preset rules to automatically update user roles when user behavior changes significantly; determine access rights based on user attributes (such as position, department), environmental factors (such as device type, network location) and resource attributes (such as data sensitivity). Combined with risk assessment models, permissions are dynamically adjusted according to the current security threat level. Instantly assess the risk of each access request and adjust permissions accordingly. Regularly re-evaluate user permissions to ensure that permissions always match current business needs and security conditions.

[0036] Dynamic role and permission management can more effectively prevent unauthorized access, automatically adjust roles and permissions according to actual conditions, and respond to changes more flexibly. It reduces the workload of manually adjusting permissions and reduces the risk of human error. It helps ensure that enterprises comply with relevant laws, regulations and industry standards.

[0037] Data lineage records specifically include: using data integration tools to automatically scan various data sources within the enterprise, establishing connections with each data source, and ensuring that metadata information can be read and written; extracting metadata information from each data source, including table structure, field description, and data type, storing the extracted metadata in a central metadata warehouse, and building a data lineage map to display the source, conversion process, and usage of the data; using NLP and machine learning technology to automatically analyze the content of the data, identify sensitive data and key business data, and automatically generate data labels based on the analysis results; analyzing the dependencies between data sets to determine the impact of changes in a certain data set on other data sets; monitoring the flow of data between different systems in real time and recording detailed log information; using machine learning models to detect abnormal data flow behavior and issue timely warnings.

[0038] Data lineage records are an important part of enterprise data governance. They help ensure data quality, consistency, and compliance by tracking the entire process from the source to the end use. Deploy data integration tools to automatically scan various data sources within the enterprise. Establish connections with each data source to ensure that metadata information can be read and written. Deploy the data integration tool Apache NiFi to automatically scan various data sources within the enterprise (databases, file systems, cloud storage). Establish connections with each data source to ensure that metadata information can be read and written, using the metadata management platform Collibra (such as Apache Atlas, Collibra, Alatio). Extract metadata information from each data source, including table structure, field description, data type, etc. Store the extracted metadata in the central metadata warehouse for unified management and query. Use the DAG algorithm to build the data flow path to ensure the order and dependency of data flow. Use the syntax analyzer to parse SQL queries and extract table names, field names, and operation types. Use the ETL job parser to parse the ETL job script to extract data sources, targets, and transformation logic. Analyze the flow path of data between different systems by parsing SQL queries and ETL jobs. Based on the results of data flow analysis, a data lineage map is constructed to show the source, conversion process and usage of data. Data lineage records can effectively realize data lineage records and ensure the traceability and transparency of data throughout the life cycle. This not only improves the effectiveness and efficiency of data governance, but also provides a reliable data foundation for enterprises.

[0039] Automated Data Discovery and Classification Automated data discovery and classification uses NLP techniques to parse text data, extract keywords, entities, and semantic information, perform feature extraction and analysis on unstructured data such as images, audio, and video, trains a data classification model using machine learning algorithms, classifies according to data features, and automatically generates data labels based on the results of the classification model for subsequent data management and use; uses regular expressions, pattern matching, and machine learning algorithms to identify sensitive data, marks the identified sensitive data, and records its location and access situation, defines data quality rules and metrics, automatically evaluates data quality according to the defined rules, and generates a data quality report.

[0040] When using NLP techniques to parse text data, the text is segmented into words or phrases. Words are reduced to their basic form. Common meaningless words such as "de", "shi", etc. are removed. When extracting keywords, TF-IDF is used to measure the importance of a word in a document. A pre-trained model BERT is used to identify entities in the text, such as person names, organization names, etc. When extracting semantic information, methods such as LDA are used to discover potential topics in a document collection. A supervised learning model SVM is used to judge the sentiment tendency of the text. A pre-trained model ResNet is used to extract image features, Mel-frequency cepstral coefficients are used for audio feature extraction, and GRU is used to process time series data to capture long-term dependencies in videos for video feature extraction.

[0041] Perform feature extraction and analysis on unstructured data such as images, audio, and video, train a data classification model using machine learning algorithms, classify according to data features, and automatically generate data labels based on the results of the classification model for subsequent data management and use. Select appropriate algorithms. The following are the processing steps for audio data:

[0042] 1) Extract Mel-frequency cepstral coefficient features. Divide the audio signal into multiple frames. Perform Fourier transform on each frame to obtain the spectrum. Pass the spectrum through a set of Mel filters, each filter covering a Mel frequency range. Calculate the energy of each filter. Take the logarithm of the energy of each filter to compress the dynamic range. Perform DCT transform on the logarithmic energy to obtain the cepstral coefficients. Usually, only the first few (such as 13) cepstral coefficients are retained as MFCC features.

[0043] 2) Select the classification algorithm K-means and train the model.

[0044] 3) Evaluate the model performance using methods such as k-fold cross-validation.

[0045] 4) Automatically generate audio labels according to the classification results.

[0046] Use regular expressions, pattern matching, and machine learning algorithms to identify sensitive data, mark the identified sensitive data, and record its location and access. Define data quality rules and indicators, automatically evaluate data quality based on the defined rules, and generate data quality reports.

[0047] 1) Determine the types of sensitive data that need to be identified, such as corporate confidential data.

[0048] 2) Write a corresponding regular expression for each sensitive data type.

[0049] 3) Use a regular expression library (such as Python's re module) for pattern matching.

[0050] 4) Add markers (such as HTML tags, special characters, etc.) around the identified sensitive data.

[0051] 5) Record the specific location of sensitive data in the document (such as line number, column number). Record access logs of sensitive data, including access time, user ID, etc.

[0052] 6) Check data according to defined rules and calculate various data quality indicators according to formulas.

[0053] Data security management adds tags to sensitive data, encrypts sensitive data during data processing, anonymizes identity information, uses machine learning models to detect abnormal data access behavior, and implements instant data backup through LVMSnapshot technology.

[0054] Encrypt sensitive data during data processing, including: selecting the RSA encryption algorithm and generating the key required for encryption. Encrypt sensitive data during data processing. Decrypt the encrypted data when needed.

[0055] Using machine learning models to detect abnormal data access behaviors, the steps include: Collecting access logs, including timestamps, user IDs, access resources, etc. Extracting features from access logs, such as access frequency, access time, access resource type, etc. Using the unsupervised learning algorithm Isolation Forest to train anomaly detection models. And using the trained model to perform anomaly detection and issue warnings in a timely manner.

[0056] Use LVM Snapshot technology to achieve instant data backup, create and configure logical volumes. Enable the snapshot function of logical volumes. Create snapshots of logical volumes regularly or on demand. Restore data through snapshots when needed.

[0057] In step S2, it specifically includes: when establishing a hierarchically managed data center, classifying the data according to the frequency of use of the data and configuring the corresponding access speed.

[0058] Define data classification standards:

[0059] Hot data: Frequently accessed data, typically used for real-time analysis and transaction processing.

[0060] Warm data: Data that is accessed occasionally, perhaps for periodic reporting or analysis.

[0061] Cold data: Data that is rarely accessed, usually used for archiving and historical records. Use log analysis tools (such as ELKStack) to monitor and analyze data access patterns. Define a lifecycle management strategy for data from hot data to cold data, and set up a cache mechanism for hot data to improve access speed.

[0062] In step S3, first, a machine learning algorithm is used to train a model to predict key indicators of data governance, including data quality, data security, and data retrieval speed.

[0063] Data quality includes checking the completeness, consistency, accuracy, and timeliness of data.

[0064] Data security includes the risk of sensitive data leakage and the security of data encryption algorithms.

[0065] Data retrieval speed includes query response time, concurrent query processing capability, and indexing efficiency.

[0066] Secondly, based on the data analysis results, an adaptive data governance strategy is generated. The data governance strategy includes automated data cleaning rules, security vulnerability detection and repair strategies, data lifecycle management strategies, and access rights adjustment strategies.

[0067] Automated data cleaning rules include missing value processing, outlier detection, and duplicate record processing.

[0068] Security vulnerability detection and repair strategies use threat intelligence platforms (AlienVault OTX) to obtain the latest threat information. Develop scripts or use automation tools (Ansible, Chef) to repair vulnerabilities.

[0069] Data lifecycle management policies define the various stages of data from creation to archiving or deletion. Develop data migration strategies between different storage tiers based on the frequency of data use. Use data lifecycle management tools (AWS S3LifecyclePolici es) to automatically migrate data.

[0070] Access rights adjustment strategy, collect and analyze user access logs, identify abnormal access patterns, dynamically adjust access rights based on users' actual access needs and behaviors, and generate reports.

[0071] Finally, apply adaptive data governance strategies to data governance, establish a feedback mechanism, collect feedback and logs, and continuously improve data governance strategies.

[0072] When establishing a feedback mechanism, collect feedback first: Collect feedback from users, system administrators, and other stakeholders. Record the system's operation logs, including operation logs, security logs, performance logs, etc. Use statistical analysis to analyze feedback and logs. Adjust data governance strategies based on the analysis results. Repeat the above process to form a closed-loop feedback mechanism.

[0073] The present invention realizes intelligent role and authority management, comprehensive data lineage records, automated data discovery and classification, enhanced data security measures, and can continuously optimize governance strategies based on real-time data analysis results, thereby improving the efficiency, accuracy and security of data governance.

[0074] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sensitive data governance method based on big data analysis, characterized by: The steps include: S1. Establish a dynamic data governance framework, including dynamic role and permission management, data lineage records, automated data discovery and classification, and data security management; S2. Establish a hierarchical management data center to achieve standardized data management; S3, adaptive data governance strategy, builds a data governance strategy based on big data analysis for the dynamic data governance framework.

2. The sensitive data management method based on big data analysis according to claim 1 is characterized by: In step S1, dynamic role and permission management includes role definition and permission allocation. When defining roles, roles and permissions are automatically adjusted according to user behavior and context; when allocating permissions, an adaptive access control mechanism is used to dynamically adjust permissions according to user attributes, environmental factors and business needs.

3. The sensitive data management method based on big data analysis according to claim 1 is characterized by: In step S1, data lineage records specifically include: using data integration tools to automatically scan various data sources within the enterprise, establishing connections with each data source, and ensuring that metadata information can be read and written; extracting metadata information from each data source, including table structure, field description, and data type, storing the extracted metadata in a central metadata warehouse, and building a data lineage map to display the source, conversion process, and usage of the data; using NLP and machine learning technology to automatically analyze the content of the data, identify sensitive data and key business data, and automatically generate data labels based on the analysis results; analyzing the dependencies between data sets to determine the impact of changes in a certain data set on other data sets; monitoring the flow of data between different systems in real time and recording detailed log information; using machine learning models to detect abnormal data flow behavior and issue early warnings in a timely manner.

4. The sensitive data management method based on big data analysis according to claim 1 is characterized by: In step S1, automated data discovery and classification uses NLP technology to parse text data, extract keywords, entities and semantic information, perform feature extraction and analysis on unstructured data such as images, audio and video, use machine learning algorithms to train data classification models, classify data according to data features, and automatically generate data labels based on the results of the classification model to facilitate subsequent data management and use; use regular expressions, pattern matching and machine learning algorithms to identify sensitive data, mark the identified sensitive data, and record its location and access status, define data quality rules and indicators, automatically evaluate data quality according to the defined rules, and generate a data quality report.

5. The sensitive data management method based on big data analysis according to claim 4 is characterized by: NLP technology is used to parse text data, segment text into words or phrases, restore words to their basic form, and remove meaningless words; when extracting keywords, TF-IDF is used to measure the importance of a word in a document, and the pre-trained model BERT is used to identify entities in the text; when extracting semantic information, the LDA method is used to discover potential topics in a document collection, the supervised learning model SVM is used to determine the sentiment tendency of the text, the pre-trained model ResNet is used to extract image features, the Mel-frequency cepstral coefficients are used for audio feature extraction, and GRU is used to process time series data to capture long-term dependencies in the video and extract video features.

6. The sensitive data management method based on big data analysis according to claim 1 is characterized by: In step S1, data security management adds tags to sensitive data, encrypts sensitive data during data processing, anonymizes identity information, uses machine learning models to detect abnormal data access behavior, and implements instant data backup through LVM Snapshot technology.

7. The sensitive data management method based on big data analysis according to claim 1 is characterized by: In step S2, it specifically includes: when establishing a hierarchically managed data center, classify the data according to the frequency of use of the data and configure the corresponding access speed; define data classification standards, hot data: frequently accessed data, used for real-time analysis and transaction processing; warm data: occasionally accessed data, used for regular reporting or analysis; cold data: rarely accessed data, used for archiving and historical records; use log analysis tools to monitor and analyze data access patterns, define data lifecycle management strategies from hot data to cold data, and set up a cache mechanism for hot data.

8. The sensitive data management method based on big data analysis according to claim 1 is characterized by: In step S3, it includes: 31) Using machine learning algorithms to train models to predict key indicators of data governance, including data quality, data security, and data retrieval speed; 32) Generate adaptive data governance strategies based on data analysis results. The data governance strategies include automated data cleaning rules, security vulnerability detection and repair strategies, data lifecycle management strategies, and access rights adjustment strategies; 33) Apply adaptive data governance strategies to data governance, establish feedback mechanisms, collect feedback and logs, and continuously improve data governance strategies.

9. The sensitive data management method based on big data analysis according to claim 8 is characterized by: In step 32), the automated data cleaning rules include missing value processing, outlier detection, and duplicate record processing; the security vulnerability detection and repair strategy uses the threat intelligence platform to obtain the latest threat information, develop scripts or use automated tools to repair vulnerabilities; the data lifecycle management strategy defines the various stages of data from creation to archiving or deletion, formulates data migration strategies between different storage layers based on the frequency of data use, and uses data lifecycle management tools to automatically migrate data; Access permission adjustment strategy, collect and analyze user access logs, identify abnormal access patterns, dynamically adjust their access permissions based on their actual access needs and behaviors, and generate reports.

Images