Data management method based on big data classification

Abstract

The invention discloses a data governance method based on big data classification, which comprises the following steps: identifying keywords and phrases in data, carrying out self-learning on the data by utilizing a machine learning algorithm, classifying the data, and adding corresponding labels to the data according to a classification result; setting access authority and login authentication for each user, and recording access behaviors and operation logs of the users; carrying out encryption processing on the sensitive data; carrying out desensitization processing on the sensitive data, and adopting a data integrity verification mechanism in a data transmission process; the abnormal condition in the data is monitored, the data flow is monitored, and the machine learning technology is utilized to prevent data leakage. According to the method, sensitive data can be more effectively protected, and the safety and efficiency of data management are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data governance, and in particular relates to a data governance method based on big data classification. Background Art

[0002] With the rapid development of information technology and the widespread application of the Internet, the era of big data has arrived. In this era, massive amounts of data are generated, collected and analyzed, providing unprecedented opportunities for corporate decision-making, scientific research and social management. However, these data contain a large amount of sensitive information, including the company's core technical data, business secrets and personal privacy data. Once these sensitive data are leaked, they will not only cause serious economic losses to companies and individuals.

[0003] Existing data governance methods have many deficiencies in data identification and classification, access control and authority management, data encryption and desensitization, real-time detection and early warning, and are unable to meet the security needs of the modern big data environment. The deficiencies are manifested in:

[0004] 1. Existing methods rely on predefined rules, lack self-learning and adaptive capabilities, and cannot handle complex and changing data.

[0005] 2. The traditional RBAC mechanism is too rigid and difficult to adapt to a dynamically changing environment; user behavior monitoring and logging are not detailed enough.

[0006] 3. Traditional encryption and desensitization technologies pose security risks and may undermine data availability.

[0007] 4. The existing anomaly detection system is rule-based and cannot effectively identify unknown attack patterns; the real-time monitoring capability is insufficient.

[0008] 5) The application of machine learning in data governance is insufficient, and there is a lack of effective model updating mechanism. Summary of the invention

[0009] In order to solve the shortcomings of existing data governance methods in data identification and classification, access control and authority management, data encryption and desensitization, real-time detection and early warning, the purpose of the present invention is to provide a data governance method based on big data classification, which can more effectively protect sensitive data and improve the security and efficiency of data governance by introducing advanced machine learning technology and encryption algorithms.

[0010] In order to achieve the purpose of the present invention, the technical solution adopted is:

[0011] A data governance method based on big data classification includes the following steps:

[0012] S1. Data identification and classification: Identify keywords and phrases in the data, use machine learning algorithms to self-learn the data, classify the data, and add corresponding labels to the data based on the classification results;

[0013] S2. Access control and authority management: set access rights and login authentication for each user, and record the user's access behavior and operation log;

[0014] S3, Data encryption and desensitization: Encrypt sensitive data; desensitize sensitive data, and use data integrity verification mechanism during data transmission;

[0015] S4. Real-time detection and early warning: Monitor anomalies in data, monitor data traffic, and use machine learning technology to prevent data leakage.

[0016] As an optimization solution of the present invention, in step S1, keywords and phrases are identified by the BERT model, and the data is self-learned by the machine learning algorithm, and the data is classified specifically as follows:

[0017] 11) Collect data, extract features, and preprocess the data;

[0018] 12) Select support vector machine SVM as the initial classifier, select regularization parameter C and learning rate η, and use historical data sets to perform preliminary training on SVM;

[0019] 13) Receive new data and pre-process the new data;

[0020] 14) After receiving a certain amount of data, the SVM model update is triggered. For each new batch of data, the SVM parameters are updated using stochastic gradient descent;

[0021] 15) Track the performance changes of the model over time to detect overfitting or underfitting. If the model accuracy decreases, reduce the learning rate η. If the model is overfitting, increase the regularization parameter C. If the model is underfitting, reduce the regularization parameter C.

[0022] As an optimization solution of the present invention, in step S2, new users are allowed to create accounts; the username and password are verified and a session token is generated; different roles are assigned to users, and the permissions of different roles are defined. When users try to access resources, it is checked whether they have the corresponding permissions; and each access and operation of the user is recorded, including timestamp, user ID, operation type and result.

[0023] As an optimization solution of the present invention, when encrypting sensitive data, the Paillier homomorphic encryption algorithm is used, specifically generating a public key and a private key; encrypting data using the public key; performing addition or multiplication operations on the ciphertext; and decrypting data using the private key. The Paillier homomorphic encryption formula is:

[0024] Decrypt(Encrypt(a)·Encrypt(b))=a+b(modn)

[0025] Among them: a is the first plaintext data; b is the second plaintext data.

[0026] As an optimization solution of the present invention, when desensitizing sensitive data, the sensitivity is first defined, and then noise that conforms to the Laplace distribution is generated; finally, the noise is added to the original data.

[0027] As an optimization solution of the present invention, in step S4, a random forest model is used to perform anomaly detection on the labeled data, and the specific steps are as follows:

[0028] 41) Divide the data set into a training set and a test set, use the training set to train the random forest classifier, and use the cross-validation method to optimize the hyperparameters of the random forest model;

[0029] 42) Use the trained random forest model to predict the test set and then calculate the evaluation indicators, including accuracy, recall, and F1 score evaluation indicators, to evaluate the performance of the random forest model.

[0030] As an optimization solution of the present invention, the machine learning technology is used to prevent data leakage as follows:

[0031] 1) Segment the data into time windows, with each window as a sample, and use sliding windows to generate multiple time series samples;

[0032] 2) The LSTM classifier is used to classify time series samples and determine whether a time series has data leakage.

[0033] The present invention identifies keywords and phrases in the data, uses machine learning algorithms to self-learn the data, classifies the data, and adds corresponding labels to the data based on the classification results; sets access rights and login authentication for each user, and records the user's access behavior and operation logs; encrypts sensitive data; desensitizes sensitive data, and adopts a data integrity verification mechanism during data transmission; monitors abnormal conditions in the data, monitors data traffic, and uses machine learning technology to prevent data leakage.

[0034] The present invention has the following positive effects:

[0035] 1. The present invention uses a self-learning mechanism combining the BERT model and SVM to more accurately identify and classify data, reduce misclassification and missed classification, and improve the efficiency of data management;

[0036] 2. The present invention introduces dynamic role allocation and detailed user behavior monitoring to make access control more flexible and sophisticated, and can better adapt to complex business needs and security environments;

[0037] 3. The present invention uses the Paillier homomorphic encryption algorithm and Laplace noise desensitization technology, which not only improves the security of data, but also maintains the availability of data and prevents data leakage and abuse;

[0038] 4. The present invention uses machine learning models such as random forest and LSTM to monitor data traffic and user behavior in real time, promptly discover and respond to potential data leakage incidents, and reduce security risks;

[0039] 5. The present invention ensures that the machine learning model can adapt to new data and threat patterns and maintain long-term effectiveness and robustness through continuous model updates and parameter adjustments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0042] like Figure 1 As shown, a data governance method based on big data classification includes the following steps:

[0043] S1. Data identification and classification: Identify keywords and phrases in the data, use machine learning algorithms to self-learn the data, classify the data, and add corresponding labels to the data based on the classification results;

[0044] In step S1, the BERT model is used to identify keywords and phrases. The BERT model can improve the recognition accuracy of keywords and phrases and better understand the context. The machine learning algorithm is used to self-learn the data and classify the data into:

[0045] 1) Collect data, extract features, and preprocess the data; preprocessing includes cleaning text data (removing punctuation, stop words, etc.). Convert text to feature vectors. Standardize or normalize feature vectors

[0046] 2) Select support vector machine SVM as the initial classifier, select regularization parameter C and learning rate η, and use historical data sets to perform preliminary training on SVM;

[0047] The goal of SVM is to find an optimal hyperplane to maximize the interval between two categories. For a given data point (xi, yi), is the input feature vector, y i ∈{-1,+1} is the corresponding label, and the formula is expressed as:

[0048]

[0049] sty i (w·φ(x i )+b)≥1-ξ i ,ξ i ≥0

[0050] w is the weight vector, b is the bias term, and φ(x) is the function that maps to the high-dimensional space. ξ i is a slack variable that allows for a certain degree of misclassification. C is a regularization parameter that controls the tradeoff between maximizing the margin and minimizing misclassification. C controls the tradeoff between maximizing the margin and minimizing misclassification. You can start with a larger range and use an SGD-based method, which requires setting a learning rate. Usually you can start with a fixed learning rate.

[0051] 3) Receive new data and pre-process the new data;

[0052] 4) After receiving a certain amount of data, the SVM model is updated. For each new batch of data, the SVM parameters are updated using stochastic gradient descent (SGD). For each newly received data point (x i ,y i ), use the following update rule to adjust w and b, if y i (w·x i +b)<1, that is, the sample violates the boundary condition, and the parameters need to be updated:

[0053] ;

[0054] b←b+ηy i ;λ=1 / (Cn), where n is the number of training samples, and λ is actually

[0055] The coefficient of the regularization term. If y i (w·x i +b)≥1, no parameter update is required.

[0056] 5) Track the performance of the model over time to detect overfitting or underfitting. If the model accuracy drops, reduce the learning rate η to adjust the model parameters more finely. If the model is overfitting, increase the regularization parameter C. If the model is underfitting, reduce the regularization parameter C. Exponential decay and step size decay can be used to automatically adjust the learning rate.

[0057] S2. Access control and authority management: set access rights and login authentication for each user, and record the user's access behavior and operation log;

[0058] In step S2, new users are allowed to create accounts; the username and password are verified and a session token is generated; different roles (administrator, ordinary user) are assigned to users, and the permissions of different roles (read, write, delete) are defined. When a user tries to access a resource, it is checked whether the user has the corresponding permissions; each access and operation of the user is recorded, including timestamp, user ID, operation type and result.

[0059] S3, Data encryption and desensitization: Encrypt sensitive data; desensitize sensitive data, and use data integrity verification mechanism during data transmission;

[0060] When encrypting sensitive data, the Paillier homomorphic encryption algorithm is used to process the ciphertext, and the decrypted result is consistent with the result of the same operation on the plaintext. Specifically, the public key and private key are generated; the public key is used to encrypt the data; addition or multiplication operations are performed on the ciphertext; and the private key is used to decrypt the data. The Paillier homomorphic encryption formula is:

[0061] Decrypt(Encrypt(a)·Encrypt(b))=a+b(modn)

[0062] Among them: a is the first plaintext data; b is the second plaintext data, · represents modulo n 2 The following multiplication operation.

[0063] Encrypt(a): The ciphertext after a is encrypted using the public key.

[0064] Encrypt(b): The ciphertext after b is encrypted using the public key.

[0065] Encrypt(a)·Encrypt(b): two ciphertexts are in modulus n 2 The following multiplication operation.

[0066] Decrypt(Encrypt(a)·Encrypt(b)): Decrypt the above multiplication result, and the result obtained is a+b(modn).

[0067] When desensitizing sensitive data, differential privacy protects the privacy of individuals by adding random noise. First, the sensitivity is defined, and then the noise is generated that conforms to the Laplace distribution; finally, the noise is added to the original data.

[0068] Noisy Data=Original Data+Laplace Noise

[0069] Noisy Data: Data after adding noise.

[0070] Original Data: Original data.

[0071] Laplace Noise: Noise drawn from a Laplace distribution.

[0072]

[0073] Sensitivity: The sensitivity of a function, defined as the maximum change in the function on adjacent data sets. For a query function f, the sensitivity Δf is defined as:

[0074] Where D and D′ are two adjacent datasets. v Privacy budget (or privacy loss parameter) controls the degree of privacy protection.

[0075] S4. Real-time detection and early warning: Monitor anomalies in data, monitor data traffic, and use machine learning technology to prevent data leakage.

[0076] In step S4, the random forest model is used to perform anomaly detection on the labeled data. The specific steps are as follows:

[0077] A. Divide the dataset into training and test sets, use the training set to train the random forest classifier, and use cross-validation to optimize the hyperparameters of the random forest model; set the hyperparameter grid. Use GridSearchCV for cross-validation.

[0078] B. Use the trained random forest model to predict the test set, and then calculate the evaluation indicators, including accuracy, recall, and F1 score evaluation indicators to evaluate the performance of the random forest model.

[0079] Using machine learning technology to prevent data leakage is as follows:

[0080] a. Segment the data by time window, with each window as a sample, and use sliding windows to generate multiple time series samples; use the Apache Kafka stream processing framework. Through automated and intelligent data governance processes, the need for manual intervention is reduced, management costs and complexity are reduced, and overall maintainability is improved.

[0081] b. LSTM classifier is used to classify time series samples and determine whether a time series is a data leak. LSTM classifier can classify and predict new time series samples to determine which category they belong to (such as normal behavior or abnormal behavior). LSTM classifier can effectively utilize the time dependency and patterns in time series data to improve the accuracy and reliability of anomaly detection.

[0082] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data governance method based on big data classification, characterized by: The steps include: S1. Data identification and classification: Identify keywords and phrases in the data, use machine learning algorithms to self-learn the data, classify the data, and add corresponding labels to the data based on the classification results; S2. Access control and authority management: set access rights and login authentication for each user, and record the user's access behavior and operation log; S3, data encryption and desensitization: encrypt sensitive data; Desensitize sensitive data and use a data integrity verification mechanism during data transmission; S4. Real-time detection and early warning: Monitor anomalies in data, monitor data traffic, and use machine learning technology to prevent data leakage.

2. According to the data management method based on big data classification according to claim 1, it is characterized by: In step S1, the BERT model is used to identify keywords and phrases, and the machine learning algorithm is used to self-learn the data, and the data is classified into the following categories: 11) Collect data, extract features, and preprocess the data; 12) Select support vector machine SVM as the initial classifier, select regularization parameter C and learning rate η, and use historical data sets to perform preliminary training on SVM; 13) Receive new data and pre-process the new data; 14) After receiving a certain amount of data, the SVM model update is triggered. For each new batch of data, the SVM parameters are updated using stochastic gradient descent; 15) Track the performance changes of the model over time to detect overfitting or underfitting. If the model accuracy decreases, reduce the learning rate η. If the model is overfitting, increase the regularization parameter C. If the model is underfitting, reduce the regularization parameter C.

3. The data governance method based on big data classification according to claim 2 is characterized by: In step 12), the goal of SVM is to find an optimal hyperplane to maximize the margin between the two classes, for a given data point (x i ,y i ),in, is the input feature vector, y i ∈{-1,+1} is the corresponding label, and the formula is expressed as: style i (ω·φ(x i )+b)≥1-ξ i ,x i ≥0 w is the weight vector, b is the bias term, φ(x) is the function mapped to the high-dimensional space, ξ i is the slack variable, C is the regularization parameter, which controls the trade-off between maximizing the margin and minimizing the misclassification; start with a larger range, use an SGD-based method, set the learning rate, and start with a fixed learning rate.

4. The data governance method based on big data classification according to claim 3 is characterized by: Trigger the SVM model update, and for each new batch of data, use stochastic gradient descent to update the SVM parameters; for each newly received data point (x i ,y i ), use the following update rule to adjust w and b, if y i (w·x i +b)<1, that is, the sample violates the boundary condition, and the parameters need to be updated: w←w+η(y i x i -λw); b←b+ηy i ;λ=1 / (Cn), where n is the number of training samples and λ is the coefficient of the regularization term. If y i (w·x i +b)≥1, no parameter update is required.

5. The data governance method based on big data classification according to claim 1 is characterized by: In step S2, new users are allowed to create accounts; the username and password are verified and a session token is generated; different roles are assigned to users and the permissions of different roles are defined. When users try to access resources, it is checked whether they have the corresponding permissions; each access and operation of the user is recorded, including timestamp, user ID, operation type and result.

6. The data governance method based on big data classification according to claim 1 is characterized by: In step S3, when encrypting sensitive data, the Paillier homomorphic encryption algorithm is used, specifically generating a public key and a private key; encrypting data using the public key; performing addition or multiplication operations on the ciphertext; and decrypting data using the private key. The Paillier homomorphic encryption formula is: Decrypt(Encrypt(a)·Encrypt(b))=a+b(modn) Among them: a is the first plaintext data; b is the second plaintext data.

7. The data governance method based on big data classification according to claim 1 is characterized by: In step S3, when desensitizing sensitive data, the sensitivity is first defined, and then noise conforming to the Laplace distribution is generated; finally, the noise is added to the original data.

8. The data governance method based on big data classification according to claim 1 is characterized by: In step S4, the random forest model is used to perform anomaly detection on the labeled data. The specific steps are as follows: 41) Divide the data set into a training set and a test set, use the training set to train the random forest classifier, and use the cross-validation method to optimize the hyperparameters of the random forest model; 42) Use the trained random forest model to predict the test set and then calculate the evaluation indicators, including accuracy, recall, and F1 score evaluation indicators, to evaluate the performance of the random forest model.

9. The data governance method based on big data classification according to claim 1 is characterized by: In step S4, machine learning technology is used to prevent data leakage by: 1) Segment the data into time windows, with each window as a sample, and use sliding windows to generate multiple time series samples; 2) The LSTM classifier is used to classify time series samples and determine whether a time series has data leakage.

Images