Self-adaptive static data desensitization method and device based on artificial intelligence

Abstract

The embodiment of the invention provides a self-adaptive static data desensitization method and device based on artificial intelligence, and relates to the technical field of static data desensitization, and the method comprises the steps: obtaining static data, and carrying out the preprocessing of the data; based on a bidirectional long-short-term memory network model and an attention mechanism, semantic and context features are extracted from the preprocessed static data; performing classification decision on the static data through a multi-layer perceptron model, and judging whether each data fragment contains sensitive information or not; optimizing the multi-layer perceptron model according to the screening result of the sensitive information; the data is automatically acquired, so that the comprehensiveness of acquiring the sensitive information is ensured, and the recognition precision of the sensitive information is improved; context dependence of data is fully considered, monitoring and analysis of real-time data streams are achieved, and newly appearing sensitive information is found in time. Large-scale data sets are processed and analyzed through the artificial intelligence technology, the model is optimized, and the overall processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of static data desensitization, and specifically, to an adaptive static data desensitization method and device based on artificial intelligence. Background Art

[0002] Static Data Masking (SDM) is a data security technology designed to protect sensitive data from unauthorized access and leakage risks. With the development of information technology and the surge in data volume, more and more sensitive information is stored in databases, including personal identity information, financial data, health records, etc. If this data is not properly protected, once leaked, it may lead to serious privacy violations and economic losses.

[0003] Database security has become a focus in the field of information security, especially in industries such as finance and healthcare, which have strict requirements on data confidentiality and integrity. In order to comply with relevant data protection regulations and standards, such as the EU's General Data Protection Regulation (GDPR) and China's Personal Information Protection Law (PIPL), enterprises must take effective measures to protect sensitive data.

[0004] Static data desensitization technology transforms the original data into data that looks real but is actually fictitious by deforming, shielding, replacing, randomizing or encrypting sensitive data. The processed data can be safely used in non-production environments such as development, testing and training without exposing real sensitive information. Static desensitization is usually performed before the data is moved or copied to another environment to ensure the one-time large-scale use of desensitized data.

[0005] As data security threats continue to evolve, static data desensitization technology is also constantly improving, integrating automated sensitive data discovery, rich desensitization algorithms, and flexible policy configurations to adapt to more complex data usage scenarios and improve desensitization efficiency. In addition, static desensitization technology can also maintain the original business characteristics and relevance of the data, ensuring that the desensitized data is still useful for business analysis and decision-making.

[0006] Privacy discovery methods for traditional static data desensitization technologies usually rely on predefined sensitive data patterns, such as social security numbers, credit card numbers, etc. These methods may include regular expression matching, keyword search, and expert rules. However, these methods have several limitations: manually defining sensitive data patterns requires experts to predefine which data is sensitive, which is not only time-consuming and labor-intensive, but may not cover all potential sensitive information; ignoring contextual sensitivity, some data items may not be sensitive in a specific context, but may become sensitive in other situations, and traditional methods often fail to fully consider the contextual dependence of data; it is difficult to adapt to emerging sensitive data types. As data types continue to evolve, new sensitive data may appear, and traditional methods are difficult to update in time to identify these new sensitive data; inefficient in processing large data sets. When processing large-scale data sets, traditional methods may be inefficient due to high computational complexity. Summary of the invention

[0007] The purpose of the embodiments of the present application is to provide an adaptive static data desensitization method and device based on artificial intelligence to solve the problems existing in the existing static desensitization methods.

[0008] In a first aspect, an embodiment of the present application provides an adaptive static data desensitization method based on artificial intelligence, comprising: Obtain static data and preprocess the data; Extract semantic and contextual features from preprocessed static data based on a bidirectional long short-term memory network model and attention mechanism; Use a multi-layer perceptron model to classify static data and determine whether each data segment contains sensitive information; The multi-layer perceptron model is optimized based on the screening results of sensitive information.

[0009] In the above implementation process, the embodiment of the present application obtains static data and preprocesses the data; based on the bidirectional long short-term memory network model and the attention mechanism, semantic and contextual features are extracted from the preprocessed static data; classification decisions are made on the static data through the multi-layer perceptron model to determine whether each data segment contains sensitive information; the multi-layer perceptron model is optimized according to the screening results of sensitive information; data is automatically acquired to ensure the comprehensiveness of sensitive information acquisition and improve the recognition accuracy of sensitive information; semantic and contextual features are extracted from the preprocessed static data to determine whether each data segment contains sensitive information, fully consider the contextual dependence of the data, realize real-time data flow monitoring and analysis, and timely discover newly emerging sensitive information; artificial intelligence technology processes and analyzes large-scale data sets, and optimizes the model to improve overall processing efficiency.

[0010] Furthermore, the obtaining of static data and preprocessing of the data includes: Collect input static data and merge the data; Perform quality assessment, verification and processing of static data; Perform data cleaning, word segmentation and vocabulary construction on static data.

[0011] In the above implementation process, through preprocessing, high-quality, accurately classified and labeled data can be provided for the AI-based adaptive static data desensitization method, laying a solid foundation for subsequent desensitization processing, effectively protecting data security and privacy, while retaining the data’s availability and analytical value to the greatest extent.

[0012] Furthermore, the extraction of semantic and contextual features from the preprocessed static data based on the bidirectional long short-term memory network model and the attention mechanism includes: Construct a bidirectional long short-term memory network model; Establish attention mechanism; Train and optimize the bidirectional long short-term memory network model; Extract semantic and contextual features from preprocessed static data.

[0013] In the above implementation process, the use of bidirectional long short-term memory network model and attention mechanism can effectively extract semantic and contextual features from the data, provide strong support for various natural language processing and text analysis tasks, and improve the performance and effect of the tasks.

[0014] Furthermore, the classification decision of the static data is made by using a multi-layer perceptron model to determine whether each data segment contains sensitive information, including: Clean the training data, extract data features and vectorize them; Construct an MLP multi-layer perceptron model; Train the MLP model; Evaluate and optimize the MLP model; The static data is input into the MLP model for classification decision-making to determine whether each data segment contains sensitive information.

[0015] In the above implementation process, the MLP model can effectively classify data fragments and determine whether they contain sensitive information, providing strong support for data security and privacy protection. In practical applications, appropriate adjustments and optimizations need to be made according to specific data types and task requirements to improve the performance and accuracy of the model.

[0016] Furthermore, the multi-layer perceptron model is optimized according to the classification result of the sensitive information, including: De-duplicate the multi-layer perceptron model based on the classification results of sensitive information; Merge sensitive information fragments of classification results; The classification results are further filtered according to the defined rules.

[0017] In the above implementation process, the model prediction results are optimized, such as deduplication, merging adjacent sensitive information fragments, and further screening based on rules, to improve the accuracy and practicality of the results.

[0018] Furthermore, the collecting of input static data and merging the data includes: For different types of data sources, corresponding connection interfaces and configuration modules are used to extract static data from the data source, and the extracted static data is converted; the extracted data includes full extraction and incremental extraction; Merging the extracted and converted data according to predetermined rules, and storing the merged data in a temporary data storage area; The quality assessment, verification and processing of static data include: Write a missing value detection algorithm, traverse each field in the data set, and count the number and location of missing values; Write a data accuracy detection program to traverse each record in the data set and verify it according to the set rules; Define data validation rules and constraints, and set validation constraint rules based on data type and business requirements; For data from the same field in multiple data sources or different tables, perform consistency comparison. If there is data inconsistency, analyze the cause and take measures; The data cleaning, word segmentation and vocabulary building of static data include: Write data cleaning scripts and select corresponding cleaning methods according to data types; Select a word segmentation tool based on the data type to perform word segmentation on static data; The vocabulary is constructed based on the set rules for the static data after word segmentation.

[0019] Furthermore, the construction of a bidirectional long short-term memory network model includes: Receive preprocessed training data through the input layer; Use a deep learning framework to build a bidirectional long short-term memory Bi-LSTM layer; Concatenate the forward and backward hidden states of the last time step of the Bi-LSTM layer; Wherein, establishing the attention mechanism includes: Define the attention function and calculate the attention weight; Use the attention weights to perform weighted summation on the output sequence of Bi-LSTM; The training and optimization of the bidirectional long short-term memory network model includes: Define the loss function; Select an optimization algorithm to update parameters; Divide the labeled data into training set, validation set and test set; Train the model on the training set, calculate the loss based on the input data and labels, calculate the gradient through the back-propagation algorithm and update the model parameters; Adjust the model's hyperparameters based on the performance indicators of the validation set, where the performance indicators include accuracy and F1 value; The step of extracting semantic and context features from the preprocessed static data includes: Input the preprocessed static data into the Bi-LSTM model; Extract semantic and contextual features and apply them to natural language processing and text analysis tasks based on the features.

[0020] Furthermore, the data cleaning of the training data, extracting data features and vectorizing the data includes: Select the corresponding cleaning method according to the data type to clean the training data; Select the corresponding feature extraction method according to the data type to extract and vectorize the data; The construction of the MLP multi-layer perceptron model includes: Divide the training data into training set, validation set and test set according to the set ratio; Design the model architecture, determine the number of layers, neurons, and activation functions; Define the MLP model using a deep learning framework; Wherein, the training MLP model includes: Collect annotated training data sets, where each piece of data is marked as whether it contains sensitive information; Define model parameters, where model parameters include activation function and layer size; Define loss function and optimizer; Train the MLP model using the training data by calling the loss function; Input the training data into the model, perform forward propagation calculations, and obtain the model's predicted output; Use the optimizer to perform backpropagation and calculate the gradient of the loss function with respect to the model parameters; Repeat the steps of forward propagation, loss calculation, backpropagation, and parameter update for multiple training rounds until the model converges or reaches the preset number of training times; The evaluation and optimization of the MLP model includes: Choose an evaluation metric and regularly evaluate the performance of the model on the validation set; According to the performance indicators of the validation set, adjust the model's hyperparameters to optimize the model's performance, where the hyperparameters include: learning rate, batch size, and model architecture; Use the test set to perform a final evaluation of the model, calculate the model's loss value and various performance indicators on the test set, and evaluate the model's generalization ability; Based on the evaluation results, the model with the best performance is selected and saved; The static data is input into the MLP model for classification decision to determine whether each data segment contains sensitive information, including: Select the corresponding cleaning method according to the data type to clean the data; Input the preprocessed data into the trained MLP model; Classification decision processing is performed based on the judgment result of the model to determine whether each data segment contains sensitive information; wherein the judgment result includes: calculation indicators and the generated confusion matrix, and the calculation indicators include accuracy, recall rate and F1 score.

[0021] Furthermore, deduplication of the multilayer perceptron model according to the classification result of sensitive information includes: Use a set data structure to store data segments that the model predicts contain sensitive information; When a new data segment is predicted to contain sensitive information, it is compared with the stored data segments to obtain the similarity; If the similarity between the new data segment and the stored data segment is greater than the similarity threshold, it is considered to be a duplicate sensitive information segment and deduplication is performed; The sensitive information fragments of the combined classification results include: Determine similarity relationships based on data types; Determine whether adjacent data fragments belong to the same type of sensitive information; If adjacent data fragments are judged to belong to the same type of sensitive information, they are merged; The further screening of the classification results according to the defined rules includes: Define rules to further filter the model prediction results based on business needs and the characteristics of sensitive information; Apply rules to filter data segments predicted by the model to contain sensitive information one by one; For each data segment, check whether it satisfies all hard rules; For data segments that meet the hard rules, soft rules are further applied for screening; In the process of applying rules for screening, feedback information is continuously collected to evaluate the effectiveness and accuracy of the rules; The rules are updated and maintained regularly.

[0022] In a second aspect, the embodiment of the present application provides an adaptive static data desensitization device based on artificial intelligence, comprising: A preprocessing module, used for acquiring static data and preprocessing the data; Feature extraction module, which is used to extract semantic and contextual features from preprocessed static data based on a bidirectional long short-term memory network model and an attention mechanism; The classification module is used to make classification decisions on static data through a multi-layer perceptron model to determine whether each data segment contains sensitive information; The post-processing module is used to optimize the multi-layer perceptron model according to the screening results of sensitive information.

[0023] In a third aspect, an embodiment of the present application provides an electronic device, including: A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the adaptive static data desensitization method based on artificial intelligence as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of a flow chart of an adaptive static data desensitization method based on artificial intelligence provided in an embodiment of the present application; Figure 2 It is a structural schematic diagram of an adaptive static data desensitization device based on artificial intelligence provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0027] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0028] Current static data desensitization technologies have issues such as limitations in privacy discovery, which refers to the process of identifying data items containing sensitive information in a data set. This process is critical to data security management because it helps ensure that sensitive data is not improperly used or leaked when it is not needed. With effective privacy discovery, organizations can take appropriate protective measures such as data desensitization, access control, and data encryption to comply with data protection regulations and maintain customer trust.

[0029] Privacy discovery methods for traditional static data desensitization technologies usually rely on predefined sensitive data patterns, such as social security numbers, credit card numbers, etc. These methods may include regular expression matching, keyword search, and expert rules. However, these methods have several limitations: manually defining sensitive data patterns requires experts to predefine which data is sensitive, which is not only time-consuming and labor-intensive, but may not cover all potential sensitive information; ignoring contextual sensitivity, some data items may not be sensitive in a specific context, but may become sensitive in other situations, and traditional methods often fail to fully consider the contextual dependence of data; it is difficult to adapt to emerging sensitive data types. As data types continue to evolve, new sensitive data may appear, and traditional methods are difficult to update in time to identify these new sensitive data; inefficient in processing large data sets. When processing large-scale data sets, traditional methods may be inefficient due to high computational complexity.

[0030] Please see Figure 1 , Figure 1 A schematic diagram of a flow chart of an adaptive static data desensitization method based on artificial intelligence provided in an embodiment of the present application. The adaptive static data desensitization method based on artificial intelligence includes: 100. Obtain static data and preprocess the data.

[0031] 110. Collect input static data and merge the data.

[0032] 111. For different types of data sources, corresponding connection interfaces and configuration modules are used to extract static data from the data sources, and the extracted static data is converted; wherein the extracted data includes full extraction and incremental extraction.

[0033] For example, for a relational database, use a database connection library (such as JDBC, ODBC, etc.) to establish a connection with the database and configure connection parameters such as database address, port number, user name, password, etc. For a file system, use a file reader and specify the file storage path and file name. For cloud storage, use the API provided by the cloud service provider for authentication and connection to obtain data access permissions.

[0034] For example, full extraction is suitable for the first data collection or when the entire data set needs to be reprocessed. All data in the data source is extracted at one time; incremental extraction only extracts new or modified data since the last extraction, and determines the incremental data by monitoring the log files, timestamps or version numbers of the data source.

[0035] During the data extraction process, data conversion operations are performed to meet the integration requirements. For example, if the field names in different data sources are inconsistent, field mapping and renaming are performed; for fields with incompatible data types, type conversion is performed, such as converting string type dates to date types. Data can also be formatted in a unified manner, such as unifying the format of telephone numbers into international standard formats or specific business requirements.

[0036] 112. The extracted and converted data are merged according to a predetermined rule, and the merged data is stored in a temporary data storage area.

[0037] For example, for structured data, it can be merged based on the primary key or unique identifier to ensure that the same record appears only once in the merged data set; for semi-structured and unstructured data, it can be merged and sorted based on file name, timestamp or other relevant metadata.

[0038] Exemplarily, the merged data is stored in a temporary data storage area, such as an in-memory database or a temporary file, for subsequent data quality assessment and other preprocessing operations. A data storage framework (such as Hadoop HDFS, Apache Cassandra, etc.) can be used to store large-scale data to ensure data reliability and scalability.

[0039] 120. Perform quality assessment, verification and processing of static data.

[0040] 121. Write a missing value detection algorithm, traverse each field in the data set, and count the number and location of missing values.

[0041] For example, for a numeric field, you can calculate its mean, median, mode and other statistics, and then select appropriate values ​​for filling according to the data distribution. For example, if the data follows a normal distribution, you can use the mean to fill; if the data has a skewed distribution, it may be more appropriate to use the median to fill.

[0042] For categorical fields, you can use the mode to fill missing values, that is, select the category with the highest frequency in the field as the fill value. In addition, you can also consider using machine learning algorithms to predict missing values, such as building a regression model based on random forests or decision trees to predict missing values ​​based on the values ​​of other related fields; for categorical fields, you can build a classification model for prediction and filling. After filling missing values, record the source and method of the filling value for subsequent analysis and tracing.

[0043] 122. Write a data accuracy detection program to traverse each record in the data set and verify it according to the set rules.

[0044] If the data is found not to meet the accuracy requirements, it will be marked as abnormal data, and the abnormality type and specific location will be recorded. For minor accuracy issues, automatic corrections can be made, such as converting data that does not conform to the date format into the correct format; for serious abnormal data, further investigation of the cause is required, and manual intervention may be required for correction.

[0045] 123. Define data validation rules and constraints, and set validation constraint rules based on data type and business requirements.

[0046] For example, for integer fields, set value range constraints; for date fields, define legal date formats; for email fields, use regular expressions to verify whether their formats are correct.

[0047] 124. Perform consistency comparison on data from multiple data sources or the same fields in different tables. If there is any inconsistency, analyze the cause and take measures.

[0048] For example, a hash algorithm or comparison function can be used to calculate the fingerprint or similarity of data to quickly find inconsistent data. For example, for two tables containing the same customer information, key fields such as customer name, address, and contact information can be hashed and then compared to see if the hash values ​​are consistent.

[0049] If data inconsistency is found, further analysis of the cause may be due to data entry errors, asynchronous data updates, or differences between data sources. Depending on the specific situation, data cleaning, data synchronization, or data coordination can be used to solve the consistency problem. For example, for data entry errors, manual corrections can be made or data cleaning tools can be used for batch processing; for the problem of asynchronous data updates, a data synchronization mechanism can be established to ensure data consistency between various data sources.

[0050] 130. Perform data cleaning, word segmentation and vocabulary construction on static data.

[0051] 131. Write a data cleaning script and select the corresponding cleaning method according to the data type to clean the data.

[0052] For example, a data cleaning script is written to remove HTML tags from text data using regular expression matching. For example, using Python's re module, define the following regular expression: pattern = re.compile(r'<.*?>'), and then use re.sub(pattern, '', text) to replace HTML tags in the text with empty strings.

[0053] To remove URL links, you can use a similar regular expression matching method, such as pattern = re.compile(r'https?: / / \S+|www\.\S+'), and perform a replacement operation.

[0054] Remove special characters and irrelevant punctuation, such as emoticons, non-ASCII characters, etc. You can define a character set that contains the characters you need to keep (such as letters, numbers, common punctuation, etc.), and then iterate over each character in the text and only keep the characters in the character set.

[0055] 132. Select the word segmentation tool according to the data type to perform word segmentation on static data.

[0056] For example, for Chinese text, using the Jieba word segmentation tool, first install the Jieba word segmentation library, and then use the following code to perform word segmentation: import jieba text = "This is a Chinese text that needs to be segmented." words = jieba.cut(text) segmented_text = " ".join(words) print(segmented_text) For English text, directly use the string split method to segment words based on spaces: text = "This is a text that needs to be segmented." words = text.split() print(words) 133. Build a vocabulary for the static data after word segmentation based on the set rules.

[0057] For example, the frequency of occurrence of all words after word segmentation can be calculated using Python's collections.Counter class: from collections import Counter words = ["apple", "banana", "apple", "orange", "banana", "apple"] word_counts = Counter(words) print(word_counts) To select high-frequency words to build a vocabulary, you can set a frequency threshold and select only words with a frequency higher than the threshold; then assign a unique index to each word, starting from 0 and increasing in sequence. You can use Python's dictionary to implement the mapping from vocabulary to index: vocab={word:index for index,(word,count)in enumerate(word_counts.items()) if count>threshold} Therefore, through preprocessing, high-quality, accurately classified and labeled data can be provided for the AI-based adaptive static data desensitization method, laying a solid foundation for subsequent desensitization processing, effectively protecting the security and privacy of the data, while retaining the data's availability and analytical value to the greatest extent.

[0058] 200. Based on the bidirectional long short-term memory network model and attention mechanism, semantic and contextual features are extracted from preprocessed static data.

[0059] 210. Construct a bidirectional long short-term memory network model.

[0060] 211. Receive the preprocessed training data through the input layer.

[0061] For example, to load a pre-trained word vector table, such as using GloVe word vectors, you can use Python's gensim library to load and search for word vectors: from gensim.models import KeyedVectors glove_model=KeyedVectors.load_word2vec_format('glove.6B.100d.txt',binary=False) def get_word_vector(word): if word in glove_model.vocab: return glove_model[word] else: return np.random.uniform(-0.1, 0.1, size=(100,))# If the word is not in the word vector table, initialize it randomly For each word, convert it into a word vector of a fixed dimension (such as 100 dimensions) by looking up the word vector table or using random initialization (if the word is not in the table). Convert the segmented text data into a word vector sequence as the input of Bi-LSTM: sentence = ["apple", "banana", "orange"] word_vectors = [get_word_vector(word) for word in sentence] input_sequence = np.array(word_vectors) 212. Use the deep learning framework to build a bidirectional long short-term memory Bi-LSTM layer.

[0062] For example, a deep learning framework such as TensorFlow or PyTorch is used to construct a Bi-LSTM layer. Taking TensorFlow as an example: from tensorflow.keras.layers import Bidirectional, LSTM bi_lstm_layer=Bidirectional(LSTM(units=128, return_sequences=True))(input_sequence) Here, the number of LSTM units is set to 128, and return_sequences=True means returning the output sequence of each time step so that the attention mechanism can be applied later. It can be understood that the number of LSTM units and other hyperparameters can be adjusted according to the characteristics of the data and the task requirements.

[0063] 213. Concatenate the forward and backward hidden states of the last time step of the Bi-LSTM layer.

[0064] For example, concatenate the forward and backward hidden states of the last time step of the Bi-LSTM layer. In TensorFlow, this can be achieved using the concatenate function: from tensorflow.keras.layers import Concatenate final_output = Concatenate()([bi_lstm_layer[:, -1, :128], bi_lstm_layer[:, 0, 128:]]) This concatenated vector serves as the feature representation of the entire sequence, which contains rich semantic and contextual information and will be used in subsequent attention mechanism calculations or directly used as a feature for other tasks.

[0065] 220. Establish an attention mechanism.

[0066] 221. Define the attention function and calculate the attention weight.

[0067] For example, let’s define the attention function. Taking a simple linear layer as an example, it can be implemented in TensorFlow as follows: from tensorflow.keras.layers import Dense, Activation attention_weights = Dense(units=1)(bi_lstm_layer) attention_weights = Activation('softmax')(attention_weights) Here, the output sequence of Bi-LSTM is first converted into a score through a linear layer, and then the score is normalized into attention weights using the softmax function; the attention weight of each time step represents the importance of the time step in the entire sequence.

[0068] 222. Use attention weights to perform weighted summation on the output sequence of Bi-LSTM.

[0069] For example, the attention weights are used to perform weighted summation on the output sequence of Bi-LSTM, which can be implemented in TensorFlow using the dot function: context_vector=tf.reduce_sum(tf.multiply(bi_lstm_layer, attention_weights), axis=1) Here, the output sequence of Bi-LSTM is multiplied element by element by the attention weight, and then summed over the time step dimension to obtain an attention-weighted feature representation context_vector. This feature representation pays more attention to the important parts of the sequence related to the current task, highlighting the key semantic and contextual information.

[0070] 230. Train and optimize the bidirectional long short-term memory network model.

[0071] 231. Define the loss function.

[0072] Specifically, for the classification task, if there are categories, assuming that the label is one-hot encoded, the prediction result is a probability distribution, and the true label is y_true, then the cross entropy loss function is defined as: from tensorflow.keras.losses import CategoricalCrossentropy loss_function = CategoricalCrossentropy() For regression tasks, if the predicted value is a continuous value and the true value is y_true, the mean square error loss function is defined as: from tensorflow.keras.losses import MeanSquaredError loss_function = MeanSquaredError() 232. Select an optimization algorithm to update parameters.

[0073] For example, the Adam optimization algorithm is used to update the parameters. In TensorFlow, it can be set as follows: from tensorflow.keras.optimizers import Adam optimizer = Adam(learning_rate=0.001) Among them, hyperparameters such as learning rate can be adjusted according to actual conditions. Too large a learning rate may cause the model to fail to converge, while too small a learning rate may make the training process too slow.

[0074] 233. Divide the labeled data into training set, validation set and test set.

[0075] 234. Train the model on the training set, calculate the loss based on the input data and labels, calculate the gradient through the back propagation algorithm and update the model parameters.

[0076] 235. Adjust the model's hyperparameters based on the performance indicators of the validation set, where the performance indicators include accuracy and F1 value.

[0077] For example, use the train_test_split function of the sklearn library to split (assuming that the data and labels are data and labels respectively): from sklearn.model_selection import train_test_split train_data, test_data, train_labels, test_labels = train_test_split(data, labels, test_size=0.2, random_state=42) train_data, val_data, train_labels, val_labels = train_test_split(train_data, train_labels, test_size=0.25, random_state=42) Train the model on the training set. In TensorFlow, use the model.fit function for training: model.compile(optimizer=optimizer, loss=loss_function, metrics=['accuracy'])# If it is a classification task, use 'accuracy' as the evaluation indicator history = model.fit(train_data, train_labels, epochs=50, batch_size=32, validation_data=(val_data, val_labels)) Here, the number of training rounds is set to 50 and the batch size is set to 32. During the training process, the model calculates the loss based on the input data and labels, and then calculates the gradient and updates the model parameters through the back-propagation algorithm. At the same time, it will be evaluated regularly on the validation set, and the model's hyperparameters will be adjusted based on the performance indicators of the validation set (such as accuracy, F1 value, etc.) to prevent overfitting. If the performance of the validation set does not improve in several consecutive rounds, you can consider reducing the learning rate or stopping training early.

[0078] 240. Extract semantic and contextual features from preprocessed static data.

[0079] 241. Input the preprocessed static data into the Bi-LSTM model.

[0080] Specifically, for new data, after the same preprocessing steps as the training data (data cleaning, word segmentation, conversion to word vectors, etc.), it is input into the trained model to obtain the attention-weighted feature representation, namely context_vector.

[0081] For example, in TensorFlow: new_data = ["new", "text", "sequence"] new_word_vectors = [get_word_vector(word) for word in new_data] new_input_sequence = np.array(new_word_vectors) new_bi_lstm_output=bi_lstm_layer(new_input_sequence)# Assume that bi_lstm_layer is a model layer that has been constructed new_attention_weights = Dense(units=1)(new_bi_lstm_output) new_attention_weights = Activation('softmax')(new_attention_weights) new_context_vector=tf.reduce_sum(tf.multiply(new_bi_lstm_output, new_attention_weights), axis=1) 242. Extract semantic and contextual features and apply them to natural language processing and text analysis tasks based on the features.

[0082] In the text classification task, the extracted feature new_context_vector can be input into a fully connected layer, and then the classification prediction is performed through the softmax function. In TensorFlow: from tensorflow.keras.layers import Dense fc_layer = Dense(units=C)(new_context_vector) # C is the number of categories predicted_probs = Activation('softmax')(fc_layer) predicted_class = tf.argmax(predicted_probs, axis=1) In sentiment analysis tasks, the sentiment of a text can be judged as positive, negative, or neutral based on the features. For example, a threshold can be set, and when the feature value is greater than the threshold, it is judged as positive sentiment, when it is less than the threshold, it is judged as negative sentiment, and when it is near the threshold, it is judged as neutral sentiment.

[0083] In information retrieval tasks, the extracted features can be used to calculate the similarity between texts, thereby retrieving and sorting related documents. For example, the similarity between feature vectors can be calculated using methods such as cosine similarity, and documents with high similarity to the query text can be ranked first.

[0084] As described above, the embodiments of the present application utilize a bidirectional long short-term memory network model and an attention mechanism to effectively extract semantic and contextual features from data, provide strong support for various natural language processing and text analysis tasks, and improve the performance and effect of the tasks.

[0085] 300. Use the multi-layer perceptron model to classify static data and determine whether each data segment contains sensitive information.

[0086] Optionally, collect task-related data from various data sources, such as extracting text data from a database for text classification tasks, obtaining image data from an image dataset for image classification tasks, etc. Organize the collected data to ensure that the data is in a uniform format and complete. For example, for text data, organize text from different sources into the same encoding format (such as UTF-8) and remove unnecessary spaces, line breaks, etc.; for image data, ensure the consistency of image size, color channels, etc.

[0087] 310. Clean the training data, extract data features and vectorize them.

[0088] 311. Select the corresponding cleaning method according to the data type to clean the training data.

[0089] For example, for text data, regular expressions are used to remove HTML tags, URL links, etc. For example, using Python's re module, a regular expression is defined: pattern = re.compile(r'<.*?>|https?: / / \S+|www\.\S+') Then use re.sub(pattern, '', text) to replace HTML tags and URL links in the text with empty strings, and remove special characters and irrelevant punctuation, such as emoticons, non-ASCII characters, etc. You can define a character set that contains the characters you need to keep (such as letters, numbers, common punctuation, etc.), and then iterate over each character in the text, retaining only the characters in the character set.

[0090] For other types of data, such as numerical data or image data, appropriate cleaning operations are performed based on the characteristics of the data. For example, for numerical data, outliers are removed (statistical methods such as the 3σ principle can be used); for image data, image filtering techniques such as Gaussian filtering can be used to remove noise.

[0091] 312. Select the corresponding feature extraction method according to the data type to perform feature extraction and vectorization of the data.

[0092] For example, for text data, the Bag of Words (BoW) model or word vector model (such as Word2Vec, GloVe, etc.) can be used for feature extraction and vectorization. For example, the Bag of Words model: build a vocabulary, count the frequency of occurrence of words in each data segment, and represent it as a vector. Each word in the vocabulary corresponds to a dimension of the vector. If the word appears in the data segment, the value of the dimension is the number of occurrences of the word; if the word does not appear, the value is 0. For example, assuming the vocabulary is ["apple", "banana", "orange"], and a data segment is "appleorange apple", its Bag of Words model representation is [2, 0, 1].

[0093] For example, the word vector model: use pre-trained word vectors (such as Word2Vec or GloVe) to map each word into a vector of fixed dimension (such as 300 dimensions), and then average or sum the word vectors in the data segment to obtain the vector representation of the entire data segment. For example, for the data segment "apple orange apple", suppose the word vector of "apple" is , the word vector of "orange" is , then the vector of the data segment can be represented as .

[0094] Exemplarily, for numerical data, standardization or normalization can be performed to make the data have the same scale. For example, using Z-score standardization, for each data point, its standardized value is calculated, where is the mean of the data and is the standard deviation of the data; Min-Max normalization can also be used to map the data to the [0, 1] interval and calculate the data value.

[0095] For example, for numerical data having a sequential relationship such as time series data, some statistical features such as mean, variance, maximum value, minimum value, slope, etc. can be extracted as feature vectors.

[0096] For example, for image data, image feature extraction algorithms such as SIFT (Scale Invariant Feature Transform) and HOG (Histogram of Oriented Gradients) can be used to extract image features, and then these features can be represented in vector form. Image pixel values ​​can also be directly vectorized to convert the image into a one-dimensional vector. For example, for a grayscale image, its pixel values ​​can be expanded into a one-dimensional vector in row-first order.

[0097] 320. Build an MLP multi-layer perceptron model.

[0098] 321. Divide the training data into training set, validation set and test set according to the set ratio.

[0099] For example, the preprocessed data set is divided into a training set, a validation set, and a test set using random partitioning or stratified partitioning. Generally speaking, the common partition ratio is 70% (training set), 20% (validation set), and 10% (test set), but the specific ratio can be adjusted according to the data scale and task requirements.

[0100] If the data has a class imbalance problem (i.e. the number of samples in different categories varies greatly), a stratified partitioning method can be used when dividing the data set to ensure that the proportion of each category in each subset is similar to that in the original data set to avoid the model being biased towards the majority during training.

[0101] 322. Design the model architecture, determine the number of layers, neurons and activation functions.

[0102] For example, choose a suitable model architecture based on the type of task and the characteristics of the data. Common model architectures include: Neural Network: Applicable to various types of data, it consists of input layer, hidden layer and output layer. The number of layers and neurons can be adjusted according to the complexity of the task. For example, for simple classification tasks, a neural network with fewer hidden layers and neurons can be used; for complex tasks, more hidden layers and neurons may be required.

[0103] Convolutional Neural Network (CNN): Mainly used to process data with grid structures such as images and audio, automatically extracting data features through components such as convolutional layers, pooling layers, and fully connected layers. For example, in image classification tasks, commonly used CNN architectures such as LeNet, AlexNet, VGGNet, ResNet, etc. can select a suitable architecture or make appropriate modifications based on the scale of the data and the requirements of the task.

[0104] Recurrent Neural Network (RNN): Suitable for processing sequence data, such as text, time series data, etc. It can capture sequence information and long-term dependencies in the data. Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) are improved versions of RNN, which can better handle the gradient vanishing and gradient exploding problems in long sequences. For example, in text generation tasks, LSTM or GRU can be used to model text sequences.

[0105] Take the example of using TensorFlow's Keras API to build a simple fully connected neural network. The code is as follows: from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense # Build a neural network with two hidden layers model = Sequential() model.add(Dense(128, activation='relu', input_shape=(input_dim,)))#Input layer, input_dim is the feature dimension of the input data model.add(Dense(64, activation='relu'))# The first hidden layer model.add(Dense(32, activation='relu'))# The second hidden layer model.add(Dense(output_dim, activation='sigmoid'))# Output layer, output_dim is the number of categories of output data (for binary classification problems, usually 1) Optional, determine the number of layers: MLP can contain an input layer, one or more hidden layers, and an output layer. For simple classification tasks, 2-3 hidden layers usually achieve good results, but the specific number of layers needs to be adjusted according to the complexity of the data and the difficulty of the task.

[0106] Optional, number of neurons: The number of neurons in the input layer depends on the feature dimension of the data. For example, if the bag-of-words model is used to vectorize text data, the feature dimension is usually equal to the size of the vocabulary (that is, the number of different words in the text data), and the number of neurons in the input layer corresponds to this feature dimension. The number of neurons in the hidden layer can be gradually reduced. For example, the first hidden layer can be set to C neurons, the second hidden layer can be set to C-1 neurons, and so on. The number of neurons in the output layer depends on the number of categories to be classified. If it is a binary classification problem (determining whether it contains sensitive information), the number of neurons in the output layer is one; if it is a multi-classification problem (such as determining the type of sensitive information), the number of neurons in the output layer is equal to the number of categories.

[0107] Optional, activation function: In the hidden layer, commonly used activation functions include ReLU (rectified linear unit), Sigmoid, and Tanh. The ReLU function has the advantages of simple calculation and fast convergence, and performs well in most cases. The Sigmoid function maps the input value to between 0 and 1 and is often used in the output layer of binary classification problems. The Tanh function maps the input value to between 0 and 1 and can also be used in the hidden layer in some cases.

[0108] 323. Define the MLP model using a deep learning framework.

[0109] For example, a deep learning framework such as TensorFlow or PyTorch is used to define the MLP model. Take TensorFlow as an example: import tensorflow as tf from tensorflow.keras.layers import Dense, Input # Define the input layer input_layer = Input(shape=(input_dim,))# input_dim is the feature dimension of the input data # Define hidden layers hidden_layer_1 = Dense(256, activation='relu')(input_layer) hidden_layer_2 = Dense(128, activation='relu')(hidden_layer_1) # Define the output layer output_layer = Dense(1, activation='sigmoid')(hidden_layer_2)# For binary classification problems, use sigmoid activation function # Create model model = tf.keras.Model(inputs=input_layer, outputs=output_layer) In this example, an MLP model with two hidden layers is defined. The input layer receives data with feature dimension input_dim. The hidden layers use ReLU activation function and the output layer uses Sigmoid activation function for binary classification decision.

[0110] 330. Train the MLP model.

[0111] 331. Collect labeled training data sets, where each piece of data is marked as whether it contains sensitive information.

[0112] Optionally, collect annotated data sets through internal collection and annotation, external data set procurement, crowdsourcing annotation platforms, open source data sets, etc., where each data segment is marked as whether it contains sensitive information. For example, for text data, some text segments can be manually annotated, and text containing personal identity information (such as name, ID number, etc.), financial information (such as bank account number, credit card number, etc.) can be marked as containing sensitive information, and other text can be marked as not containing sensitive information.

[0113] 332. Define model parameters, where the model parameters include activation functions and layer sizes.

[0114] Optional, common activation functions include: oReLU (Rectified Linear Unit): The formula is f(x) = max(0, x). It performs well in most cases, can effectively alleviate the gradient vanishing problem, speed up training, and is often used in hidden layers.

[0115] oSigmoid: The formula is f(x) = 1 / (1 + exp(-x)), which maps the input value to the interval [0, 1]. It is often used in the output layer of binary classification problems. The output value can be interpreted as the probability that the sample belongs to the positive class.

[0116] oTanh (Hyperbolic Tangent): The formula is f(x) = (exp(x) - exp(-x)) / (exp(x) +exp(-x)), which maps the input value to the [-1, 1] interval. It can be used for hidden layers in some cases, especially when the data is distributed in the [-1, 1] interval, which may work better.

[0117] oSoftmax: The formula is f(x_i) = exp(x_i) / sum(exp(x_j)), where i and j represent the category index. The output layer used for multi-classification problems converts the output value into the probability distribution of each category so that the sum of the probabilities of all categories is 1.

[0118] Optional, layer size settings: The number of neurons in the input layer usually depends on the feature dimension of the data. For example, if it is image data, the number of neurons in the input layer can be the number of pixels in the image multiplied by the number of channels; if it is text data, after feature extraction, the number of neurons in the input layer can be the dimension of the feature vector.

[0119] The number of neurons in the hidden layer can be gradually reduced or adjusted based on experience and experiments. Generally speaking, larger hidden layers can learn more complex features, but may also lead to overfitting; smaller hidden layers may have limited learning capabilities but are computationally cheaper. The appropriate hidden layer size can be determined by trying different configurations and evaluating model performance on the validation set.

[0120] The number of neurons in the output layer depends on the number of categories in the classification task. For binary classification problems, there is usually one neuron in the output layer, and its output value is thresholded (for example, values ​​greater than 0.5 are considered positive, and values ​​less than or equal to 0.5 are considered negative); for multi-classification problems, the number of neurons in the output layer is equal to the number of categories, and the output of each neuron represents the probability of the corresponding category.

[0121] 333. Define loss function and optimizer.

[0122] For example, define the loss function and optimizer. For binary classification problems, the commonly used loss function is Binary Cross Entropy, and the optimizer can be Adam (Adaptive Moment Estimation). In TensorFlow: model.compile(optimizer='adam',loss='binary_crossentropy', metrics=['accuracy']) 334. Train the MLP model using the training data by calling the loss function. For example, the MLP model is trained using the training data by calling the model.fit function. For example: history=model.fit(train_data,train_labels,epochs=50,batch_size=32,validation_data=(val_data, val_labels)) Here, the number of training rounds is set to 50 and the batch size is set to 32. During the training process, the model calculates the loss based on the input data and labels, and then calculates the gradient and updates the weights and biases of the model through the back propagation algorithm. After each round of training, it will be evaluated on the validation set and the performance indicators of the model (such as accuracy, loss value, etc.) will be recorded.

[0123] 335. Input the training data into the model, perform forward propagation calculation, and obtain the model's predicted output.

[0124] Optionally, input the training data into the model and perform forward propagation calculations to obtain the model's predicted output; for classification problems, the predicted output is usually the probability distribution of each sample belonging to each category. Based on the predicted output and the true label, calculate the cross entropy loss value. The loss value reflects the degree of difference between the model's prediction result and the true label.

[0125] 336. Use the optimizer for backpropagation to calculate the gradient of the loss function with respect to the model parameters.

[0126] Optionally, use the Adam optimizer to perform backpropagation to calculate the gradient of the loss function with respect to the model parameters; backpropagation is the process of calculating the gradient layer by layer from the output layer to the input layer using the chain rule. The Adam optimizer automatically adjusts the learning rate and updates the model parameters based on the gradient and the first-order and second-order moment estimates of the parameters. Specifically, the optimizer calculates the updated parameter values ​​based on the current gradient and the first-order and second-order moment estimates of the previous moment.

[0127] 337. Repeat the steps of forward propagation, loss calculation, backpropagation and parameter update for multiple training rounds until the model converges or reaches the preset number of training times.

[0128] Optionally, repeat the steps of forward propagation, loss calculation, back propagation, and parameter update for multiple training epochs until the model converges or reaches a preset number of training times. In each training epoch, the training data can be divided into multiple batches for processing to reduce memory usage and improve training efficiency.

[0129] In TensorFlow and PyTorch, you can use data loaders to implement batch processing. For example, in PyTorch, you can use the torch.utils.data.DataLoader class to load data and perform batch processing, as shown below: from torch.utils.data import DataLoader from torchvision.datasets import MNIST from torchvision.transforms import ToTensor # Load the MNIST dataset train_dataset = MNIST(root='. / data', train=True, transform=ToTensor(), download=True) train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True) # Train the model for epoch in range(num_epochs): for batch_data, batch_labels in train_loader: # Forward pass outputs = model(batch_data) # Calculate loss loss = loss_function(outputs, batch_labels) # Backward Propagation optimizer.zero_grad() loss.backward() # Parameter update optimizer.step() 340. Evaluate and optimize the MLP model.

[0130] 341. Select evaluation metrics and regularly evaluate the performance of the model on the validation set.

[0131] For example, indicators such as accuracy, recall, F1-Score, and precision are used to evaluate the performance of the model.

[0132] The accuracy rate is the ratio of the number of correctly classified data fragments to the total number of data fragments, and is calculated as: TP / N, where TP (True Positive) is the true positive example, that is, the number of data fragments that actually contain sensitive information and are predicted by the model to contain sensitive information, and N is the total number of data fragments; TN (True Negative) is the true negative example, that is, the number of data fragments that actually do not contain sensitive information and are predicted by the model to not contain sensitive information; FP (False Positive) is the false positive example, that is, the number of data fragments that actually do not contain sensitive information but are predicted by the model to contain sensitive information; FN (False Negative) is the false negative example, that is, the number of data fragments that actually contain sensitive information but are predicted by the model to not contain sensitive information.

[0133] The recall rate is the ratio of the number of correctly predicted data fragments containing sensitive information to the number of data fragments actually containing sensitive information, and is calculated as TP / (TP+FN).

[0134] The F1 value is an indicator that comprehensively considers precision and recall.

[0135] 342. According to the performance indicators of the validation set, adjust the hyperparameters of the model to optimize the performance of the model, where the hyperparameters include: learning rate, batch size, and model architecture.

[0136] For example, adjust the model's hyperparameters based on the evaluation results. If the model's performance on the validation set is not satisfactory, you can try the following adjustments: Adjust the learning rate: If the learning rate is too large, the model may not converge; if the learning rate is too small, the training process may be very slow. You can try to reduce or increase the learning rate, for example, adjust the learning rate from to or, then retrain the model and observe the performance changes.

[0137] Adjust the number of layers and neurons: Increase or decrease the number of hidden layers, or adjust the number of neurons in the hidden layer. For example, if the model is overfitting, you can reduce the number of layers or neurons; if the model is underfitting, you can increase the number of layers or neurons. Then retrain and reevaluate the model.

[0138] Try different activation functions: For example, replace the ReLU activation function with Sigmoid or Tanh, and observe the changes in model performance.

[0139] Use data enhancement technology: For certain types of data, such as image data or text data, you can use data enhancement technology to increase the diversity of the data and improve the generalization ability of the model. For example, for text data, you can perform synonym replacement, random deletion or insertion of words, and for image data, you can perform rotation, scaling, cropping, and other operations. Then retrain the model to see if the performance improves.

[0140] 343. Use the test set to perform a final evaluation of the model, calculate the model's loss value and various performance indicators on the test set, and evaluate the model's generalization ability.

[0141] Among them, the test set should be data that has never been used in the model training process to ensure the objectivity and reliability of the evaluation results.

[0142] For example, in TensorFlow and PyTorch, you can use a similar method to the validation set evaluation to input the test set data into the model and calculate the loss and performance indicators. For example, in TensorFlow, you can use the model.evaluate function to evaluate the test set as follows: test_loss, test_accuracy = model.evaluate(x_test, y_test) 344. Based on the evaluation results, select the model with the best performance and save it.

[0143] Optionally, based on the evaluation results, select the model with the best performance to save, and save the model's structure and parameters so that the model can be directly loaded and used for prediction in subsequent applications.

[0144] For example, in TensorFlow, you can use the model.save function to save the model as follows: model.save('my_model.h5') For example, in PyTorch, you can use the torch.save function to save the model's state dictionary (containing the model's parameters) as follows: torch.save(model.state_dict(), 'my_model.pth') Through the above technical solutions, the cross entropy loss function and Adam optimizer can be used to effectively train the model and improve the performance and generalization ability of the model.

[0145] 350. Input static data into the MLP model for classification decision to determine whether each data segment contains sensitive information.

[0146] 351. Select the corresponding cleaning method according to the data type to clean the data.

[0147] The data segments to be judged whether they contain sensitive information are subjected to the same preprocessing operations as the training data, including data cleaning, feature extraction, and vectorization, to ensure that the format and features of the data are consistent with the training data.

[0148] 352. Input the preprocessed data into the trained MLP model.

[0149] For example, the preprocessed data is input into the trained MLP model, and the output of the model is obtained by calling the model.predict function. For example: predictions = model.predict(new_data) For a binary classification problem, the output result is a value between 0 and 1, which indicates the probability that the data segment contains sensitive information. A threshold can be set. For example, if the predicted probability is greater than the threshold, the data segment is judged to contain sensitive information; if the predicted probability is less than or equal to the threshold, the data segment is judged to not contain sensitive information.

[0150] 353. Perform classification decision processing based on the judgment result of the model to determine whether each data segment contains sensitive information; wherein the judgment result includes: calculation indicators and the generated confusion matrix, and the calculation indicators include accuracy, recall rate and F1 score.

[0151] Optionally, calculate metrics: For example, accuracy refers to the ratio of the number of samples correctly classified by the model to the total number of samples. It is an overall evaluation indicator that can reflect the performance of the model on all samples. The calculation formula is: Accuracy = (TP + TN) / (TP + TN + FP + FN), where TP (True Positive) is the true positive example, that is, the number of samples that are actually positive and predicted as positive by the model; TN (True Negative) is the true negative example, that is, the number of samples that are actually negative and predicted as negative by the model; FP (False Positive) is the false positive example, that is, the number of samples that are actually negative but predicted as positive by the model; FN (False Negative) is the false negative example, that is, the number of samples that are actually positive but predicted as negative by the model.

[0152] In Python, we can use the accuracy_score function of the scikit-learn library to calculate the accuracy as follows: from sklearn.metrics import accuracy_score accuracy = accuracy_score(y_test, y_pred) For example, recall is also called recall rate, which refers to the ratio of the number of positive samples correctly classified by the model to the actual number of positive samples. It mainly focuses on the model's ability to identify positive samples. The calculation formula is: Recall = TP / (TP +FN).

[0153] In Python, recall can be calculated using the recall_score function of the scikit-learn library as follows: from sklearn.metrics import recall_score recall = recall_score(y_test, y_pred, average='binary')# For binary classification problems recall = recall_score(y_test, y_pred, average='macro')# For multi-classification problems, calculate the recall rate of each category and then take the average For example, the F1 score is the harmonic mean of precision and recall, which takes into account the accuracy and recall of the model. The higher the F1 score, the better the balance between accuracy and recall. The calculation formula is: F1=2 * (Precision * Recall) / (Precision + Recall); Among them, Precision refers to the ratio of the number of positive samples correctly classified by the model to the number of samples predicted by the model as positive. The calculation formula is Precision = TP / (TP + FP).

[0154] In Python, the F1 score can be calculated using the f1_score function of the scikit-learn library as follows: from sklearn.metrics import f1_score f1 = f1_score(y_test, y_pred, average='binary')# For binary classification problems f1 = f1_score(y_test, y_pred, average='macro')# For multi-classification problems, calculate the F1 score for each category and then take the average Optionally, generate a confusion matrix: The confusion matrix is ​​a tabular tool used to show how the model classifies samples of different categories. Its rows represent the actual categories, and its columns represent the categories predicted by the model. Each element in the confusion matrix represents the number of samples whose actual category is a certain category and predicted by the model as another category. For binary classification problems, the confusion matrix usually has four elements: true positive examples (TP), false positive examples (FP), true negative examples (TN), and false negative examples (FN). For multi-classification problems, the confusion matrix is ​​a square matrix, in which each element represents the number of samples whose actual category is a certain category and predicted by the model as another category.

[0155] In Python, you can use the confusion_matrix function of the scikit-learn library to generate a confusion matrix as follows: from sklearn.metrics import confusion_matrix cm = confusion_matrix(y_test, y_pred) The resulting confusion matrix cm is a two-dimensional array, where cm[i][j] represents the number of samples whose actual category is i and predicted by the model to be j.

[0156] To present the confusion matrix more intuitively, you can use visualization tools to visualize it. For example, you can use matplotlib and seaborn libraries to draw a heat map of the confusion matrix, as shown below: import matplotlib.pyplot as plt import seaborn as sns # Assume cm is the generated confusion matrix and class_labels is the class label list plt.figure(figsize=(8, 6)) sns.heatmap(cm,annot=True,fmt='d',cmap='Blues', xticklabels=class_labels, yticklabels=class_labels) plt.xlabel('Predicted Labels') plt.ylabel('True Labels') plt.title('Confusion Matrix') plt.show() The above code will generate an annotated confusion matrix heat map, where the number in each cell represents the corresponding number of samples. The darker the color, the more samples there are. By visualizing the confusion matrix, you can intuitively see which categories the model is prone to confusion on and which categories are correctly classified by the model at a higher rate.

[0157] Optionally, analyze the performance of the model based on the calculated accuracy, recall, F1 score and other indicators. If the accuracy is high but the recall is low, it means that the model may be too conservative and the recognition of positive samples is not comprehensive enough; if the recall is high but the accuracy is low, it means that the model may have more false positives and incorrectly classify negative samples as positive.

[0158] In addition, for multi-classification problems, you can analyze the accuracy, recall, and F1 score of each category separately to understand the performance differences of the model on different categories. If the performance of some categories is significantly lower than that of other categories, you can consider increasing the number of samples in these categories, adjusting the model structure or parameters, and other methods to improve the performance of the model on these categories.

[0159] Optionally, by observing the confusion matrix, you can understand which categories the model is easily confused between. If the confusion between some categories is serious, you can analyze the feature differences between these categories and consider adding more features or adjusting the feature extraction method to improve the model's ability to distinguish these categories.

[0160] In addition, you can also analyze the distribution of true positives, false positives, true negatives, and false negatives to understand the type of errors in the model. For example, if there are many false negatives, it means that the model may not be able to identify positive samples well. You can consider adjusting the model's threshold or increasing the weight of positive samples to improve the recall rate.

[0161] Optionally, propose corresponding improvement measures based on the analysis results of the indicators and confusion matrix. For example: Data enhancement: Increase the diversity of training data and improve the generalization ability of the model. Data enhancement techniques can be used to enhance image data, such as random rotation, flipping, cropping, scaling, etc. For text data, synonym replacement, random deletion or insertion of words can be used for enhancement.

[0162] Adjust the model structure: According to the characteristics of the data and the task requirements, adjust the model's parameters such as the number of layers, number of neurons, activation function, etc. You can try to use more complex model structures, such as deep neural networks, convolutional neural networks, etc., to improve the model's learning ability.

[0163] Tune hyperparameters: Optimize the model's hyperparameters, such as learning rate, batch size, regularization coefficient, etc. You can use grid search, random search, and other methods to find the best hyperparameter combination.

[0164] Ensemble learning: Combine multiple different models for ensemble learning to improve the performance and stability of the model. Ensemble learning methods such as bagging and boosting can be used to combine multiple weak learners into a strong learner.

[0165] For example, the data segment is processed according to the judgment result of the model. For example, if the data segment is judged to contain sensitive information, further security measures such as encryption and desensitization can be taken; if the data segment does not contain sensitive information, it can be processed or analyzed according to the normal process.

[0166] At the same time, the prediction results of the model can be explained and analyzed to understand the features on which the model makes its judgment, so as to further optimize the model and data processing flow. For example, you can view the weight distribution of the model to different features and analyze which features play a key role in judging sensitive information, so as to improve the feature extraction method or add relevant features in a targeted manner.

[0167] Therefore, the above MLP model can effectively classify data fragments and determine whether they contain sensitive information, providing strong support for data security and privacy protection. In practical applications, appropriate adjustments and optimizations need to be made according to specific data types and task requirements to improve the performance and accuracy of the model.

[0168] 400. Optimize the multi-layer perceptron model based on the screening results of sensitive information.

[0169] 410. De-duplicate the multi-layer perceptron model based on the classification results of sensitive information.

[0170] 411. Using the set data structure, the data segments predicted by the model to contain sensitive information are stored.

[0171] 412. When a new data segment is predicted to contain sensitive information, it is compared with the stored data segment to obtain similarity.

[0172] 413. If the similarity between the new data segment and the stored data segment is greater than the similarity threshold, it is considered to be a duplicate sensitive information segment and is deduplicated.

[0173] First, store the data segments that the model predicts to contain sensitive information. You can use appropriate data structures, such as lists, sets, or database tables. If the amount of data is large, you can consider using a database to store it for efficient query and operation.

[0174] When a new data segment is predicted to contain sensitive information, it is compared with the stored data segment. The appropriate comparison method can be selected according to the specific data type and characteristics. For example, for text data, the similarity of the text content can be compared; for numerical data, the size and range of the numerical value can be compared.

[0175] For example, for text data, a text similarity algorithm can be used to calculate the similarity between a new data segment and a stored data segment. Common text similarity algorithms include cosine similarity, Jaccard similarity, edit distance, etc. For example, cosine similarity is used to calculate the cosine value of the angle between two text vectors. The closer the value is to 1, the higher the similarity.

[0176] Set a similarity threshold based on the actual situation. If the similarity between the new data segment and the stored data segment exceeds this threshold, it is considered a duplicate sensitive information segment and can be deduplicated. The threshold setting needs to be adjusted according to specific business needs and data characteristics. An appropriate threshold can be determined through experiments and evaluations.

[0177] When a new data segment is determined to be duplicate data, different deduplication methods can be used. A simple method is to directly ignore the duplicate data segment without further processing. Another method is to retain a representative data segment and mark or merge the duplicate data segments. For example, for text data, the longest text segment can be selected as a representative, or duplicate text segments can be merged into a more complete text.

[0178] 420. Merge sensitive information fragments of classification results.

[0179] 421. Determine similarity relationships based on data types.

[0180] 422. Determine whether adjacent data fragments belong to the same type of sensitive information.

[0181] 423. If adjacent data fragments are judged to belong to the same type of sensitive information, they are merged.

[0182] Optionally, for ordered data, such as continuous character sequences in text data or continuous time points in time series data, the adjacent relationship can be determined based on the order of the data; for unordered data, the adjacent relationship can be defined based on certain specific rules or features. For example, for image data, adjacent sensitive areas can be determined based on the positional relationship of pixels.

[0183] For adjacent data segments, whether they belong to the same type of sensitive information can be determined based on the model's prediction results, data features, or pre-defined rules. For example, if two adjacent text segments are both predicted by the model to contain personally identifiable information and have certain similarities in content, they can be considered to belong to the same type of sensitive information.

[0184] Similarity calculation methods can be used to assist in determining the similarity of adjacent data segments. For example, for text data, the Jaccard similarity or edit distance of adjacent text segments can be calculated; for image data, the color similarity or texture similarity of adjacent regions can be calculated.

[0185] Optionally, if adjacent data segments are judged to belong to the same type of sensitive information, they can be merged. The merging strategy can be determined based on the specific data type and business requirements. For example, for text data, adjacent text segments can be connected to form a longer text; for image data, adjacent sensitive areas can be merged into a larger area.

[0186] During the merging process, attention should be paid to maintaining the integrity and accuracy of the data. For example, for text data, the merged text should maintain semantic coherence; for image data, the merged area should maintain the visual effect and information integrity of the image.

[0187] 430. Further filter the classification results according to the defined rules.

[0188] 431. Define rules to further filter the model prediction results based on business needs and the characteristics of sensitive information.

[0189] 432. Apply rules to filter data segments predicted by the model as containing sensitive information one by one.

[0190] 433. For each data segment, check whether it satisfies all hard rules.

[0191] 434. For data segments that meet the hard rules, soft rules are further applied for screening.

[0192] 435. In the process of applying rules for screening, continuously collect feedback information to evaluate the effectiveness and accuracy of the rules.

[0193] 436. Update and maintain the rules regularly.

[0194] Optionally, based on business needs and the characteristics of sensitive information, define a series of rules to further filter the results of model predictions. The rules can be formulated based on data characteristics, contextual information, business logic, etc. For example, for personal identity information, you can define rules such as "ID number must be 18 digits", "mobile phone number must be 1 digit", etc.; for financial information, you can define rules such as "bank account number must conform to a specific format", etc.

[0195] Among them, rules can be divided into hard rules and soft rules. Hard rules are rules that must be met. If a data segment does not meet the hard rules, it will be directly judged as non-sensitive information; soft rules are rules that can be adjusted according to specific circumstances and can be used to further screen and optimize results.

[0196] Optionally, apply the rules one by one to filter the data segments predicted by the model to contain sensitive information, and for each data segment, check whether it satisfies all hard rules; if it does not satisfy any hard rule, delete it from the sensitive information list.

[0197] For data fragments that meet the hard rules, soft rules can be further applied for screening. Soft rules can be adjusted according to specific business needs and data characteristics, and comprehensive evaluation can be performed by setting different weights and priorities. For example, different scores can be given according to the degree of matching between data fragments and soft rules, and then sorted and screened according to the total score.

[0198] Optionally, in the process of applying rules for screening, continuously collect feedback information to evaluate the effectiveness and accuracy of the rules. If the rules are found to be unreasonable or inaccurate, adjust and optimize them in a timely manner. The screening effect of the rules can be improved by adding new rules, modifying the conditions and parameters of existing rules, and adjusting the weights and priorities of the rules.

[0199] Update and maintain rules regularly to adapt to changing business needs and data characteristics. For example, add new rules or adjust existing rules in a timely manner as new sensitive information types emerge or business rules change.

[0200] As described above, the embodiment of the present application obtains static data and preprocesses the data; based on the bidirectional long short-term memory network model and the attention mechanism, semantic and contextual features are extracted from the preprocessed static data; classification decisions are made on the static data through a multi-layer perceptron model to determine whether each data segment contains sensitive information; the multi-layer perceptron model is optimized according to the screening results of sensitive information; data is automatically acquired to ensure the comprehensiveness of sensitive information acquisition and improve the accuracy of identifying sensitive information; semantic and contextual features are extracted from the preprocessed static data to determine whether each data segment contains sensitive information, fully consider the contextual dependence of the data, realize real-time data flow monitoring and analysis, and timely discover newly emerging sensitive information; artificial intelligence technology processes and analyzes large-scale data sets, and optimizes the model to improve overall processing efficiency.

[0201] The above steps are not to be performed in a strict order as described in the numbers, but should be understood as an overall solution.

[0202] In the second aspect, based on the above embodiments, Figure 2 A schematic diagram of the structure of an adaptive static data desensitization device based on artificial intelligence provided in an embodiment of the present application. Figure 2The artificial intelligence-based adaptive static data desensitization device provided in this embodiment specifically includes: a preprocessing module 201, a feature extraction module 202, a classification module 203 and a post-processing module 204.

[0203] Among them, the preprocessing module 201 is used to obtain static data and preprocess the data; the feature extraction module 202 is used to extract semantic and contextual features from the preprocessed static data based on the bidirectional long short-term memory network model and the attention mechanism; the classification module 203 is used to make classification decisions on the static data through the multi-layer perceptron model to determine whether each data segment contains sensitive information; the post-processing module 204 is used to optimize the multi-layer perceptron model according to the screening results of sensitive information.

[0204] As described above, the embodiment of the present application obtains static data and preprocesses the data; based on the bidirectional long short-term memory network model and the attention mechanism, semantic and contextual features are extracted from the preprocessed static data; classification decisions are made on the static data through a multi-layer perceptron model to determine whether each data segment contains sensitive information; the multi-layer perceptron model is optimized according to the screening results of sensitive information; data is automatically acquired to ensure the comprehensiveness of sensitive information acquisition and improve the accuracy of identifying sensitive information; semantic and contextual features are extracted from the preprocessed static data to determine whether each data segment contains sensitive information, fully consider the contextual dependence of the data, realize real-time data flow monitoring and analysis, and timely discover newly emerging sensitive information; artificial intelligence technology processes and analyzes large-scale data sets, and optimizes the model to improve overall processing efficiency.

[0205] The artificial intelligence-based adaptive static data desensitization device provided in the embodiment of the present application can be used to execute the artificial intelligence-based adaptive static data desensitization method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0206] On the third aspect, the embodiments of the present application also provide an electronic device, which can integrate the artificial intelligence-based adaptive static data desensitization device provided in the embodiments of the present application. Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 The electronic device includes: an input device 33, an output device 34, a memory 32 and one or more processors 31; the memory 32 is used to store one or more programs; when the one or more programs are executed by the one or more processors 31, the one or more processors 31 implement the adaptive static data desensitization method based on artificial intelligence as provided in the above embodiment. The input device 33, the output device 34, the memory 32 and the processor 31 can be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0207] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 32, thereby realizing the above-mentioned adaptive static data desensitization method based on artificial intelligence.

[0208] The electronic device provided above can be used to execute the adaptive static data desensitization method based on artificial intelligence provided in the above embodiment, and has corresponding functions and beneficial effects.

[0209] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

Claims

1. An adaptive static data desensitization method based on artificial intelligence, It is characterized in that The method comprises: Obtain static data and preprocess the data; Extract semantic and contextual features from preprocessed static data based on a bidirectional long short-term memory network model and attention mechanism; Use a multi-layer perceptron model to classify static data and determine whether each data segment contains sensitive information; The multi-layer perceptron model is optimized based on the screening results of sensitive information.

2. According to the artificial intelligence-based adaptive static data desensitization method of claim 1, It is characterized in that The step of obtaining static data and preprocessing the data includes: Collect input static data and merge the data; Perform quality assessment, verification and processing of static data; Perform data cleaning, word segmentation and vocabulary construction on static data.

3. According to the artificial intelligence-based adaptive static data desensitization method of claim 1, It is characterized in that The method extracts semantic and contextual features from preprocessed static data based on a bidirectional long short-term memory network model and an attention mechanism, including: Construct a bidirectional long short-term memory network model; Establish attention mechanism; Train and optimize the bidirectional long short-term memory network model; Extract semantic and contextual features from preprocessed static data.

4. The adaptive static data desensitization method based on artificial intelligence according to claim 1, It is characterized in that The method of making classification decisions on static data through a multi-layer perceptron model to determine whether each data segment contains sensitive information includes: Clean the training data, extract data features and vectorize them; Construct an MLP multi-layer perceptron model; Train the MLP model; Evaluate and optimize the MLP model; The static data is input into the MLP model for classification decision-making to determine whether each data segment contains sensitive information.

5. The adaptive static data desensitization method based on artificial intelligence according to claim 1, It is characterized in that The optimizing of the multi-layer perceptron model according to the classification result of the sensitive information includes: De-duplicate the multi-layer perceptron model based on the classification results of sensitive information; Merge sensitive information fragments of classification results; The classification results are further filtered according to the defined rules.

6. The adaptive static data desensitization method based on artificial intelligence according to claim 2, It is characterized in that The collecting of input static data and merging the data includes: For different types of data sources, corresponding connection interfaces and configuration modules are used to extract static data from the data source, and the extracted static data is converted; the extracted data includes full extraction and incremental extraction; Merging the extracted and converted data according to predetermined rules, and storing the merged data in a temporary data storage area; The quality assessment, verification and processing of static data include: Write a missing value detection algorithm, traverse each field in the data set, and count the number and location of missing values; Write a data accuracy detection program to traverse each record in the data set and verify it according to the set rules; Define data validation rules and constraints, and set validation constraint rules based on data type and business requirements; For data from the same field in multiple data sources or different tables, perform consistency comparison. If there is data inconsistency, analyze the cause and take measures; The data cleaning, word segmentation and vocabulary building of static data include: Write data cleaning scripts and select corresponding cleaning methods according to data types for data cleaning; Select a word segmentation tool based on the data type to perform word segmentation on static data; The vocabulary is constructed based on the set rules for the static data after word segmentation.

7. The adaptive static data desensitization method based on artificial intelligence according to claim 3, It is characterized in that The constructing of a bidirectional long short-term memory network model comprises: Receive preprocessed training data through the input layer; Use a deep learning framework to build a bidirectional long short-term memory Bi-LSTM layer; Concatenate the forward and backward hidden states of the last time step of the Bi-LSTM layer; Wherein, establishing the attention mechanism includes: Define the attention function and calculate the attention weight; Use the attention weights to perform weighted summation on the output sequence of Bi-LSTM; The training and optimization of the bidirectional long short-term memory network model includes: Define the loss function; Select an optimization algorithm to update parameters; Divide the labeled data into training set, validation set and test set; Train the model on the training set, calculate the loss based on the input data and labels, calculate the gradient through the back-propagation algorithm and update the model parameters; Adjust the model's hyperparameters based on the performance indicators of the validation set, where the performance indicators include accuracy and F1 value; The step of extracting semantic and context features from the preprocessed static data includes: Input the preprocessed static data into the Bi-LSTM model; Extract semantic and contextual features and apply them to natural language processing and text analysis tasks based on the features.

8. The adaptive static data desensitization method based on artificial intelligence according to claim 4, It is characterized in that The data cleaning of the training data, extracting the features of the data and vectorizing the data include: Select the corresponding cleaning method according to the data type to clean the training data; Select the corresponding feature extraction method according to the data type to extract and vectorize the data; The construction of the MLP multi-layer perceptron model includes: Divide the training data into training set, validation set and test set according to the set ratio; Design the model architecture, determine the number of layers, neurons, and activation functions; Define the MLP model using a deep learning framework; Wherein, the training MLP model includes: Collect annotated training data sets, where each piece of data is marked as whether it contains sensitive information; Define model parameters, where model parameters include activation function and layer size; Define loss function and optimizer; Train the MLP model using the training data by calling the loss function; Input the training data into the model, perform forward propagation calculations, and obtain the model's predicted output; Use the optimizer to perform backpropagation and calculate the gradient of the loss function with respect to the model parameters; Repeat the steps of forward propagation, loss calculation, backpropagation, and parameter update for multiple training rounds until the model converges or reaches the preset number of training times; The evaluation and optimization of the MLP model includes: Choose an evaluation metric and regularly evaluate the performance of the model on the validation set; According to the performance indicators of the validation set, adjust the model's hyperparameters to optimize the model's performance, where the hyperparameters include: learning rate, batch size, and model architecture; Use the test set to perform a final evaluation of the model, calculate the model's loss value and various performance indicators on the test set, and evaluate the model's generalization ability; Based on the evaluation results, the model with the best performance is selected and saved; The static data is input into the MLP model for classification decision to determine whether each data segment contains sensitive information, including: Select the corresponding cleaning method according to the data type to clean the data; Input the preprocessed data into the trained MLP model; Classification decision processing is performed based on the judgment result of the model to determine whether each data segment contains sensitive information; wherein the judgment result includes: calculation indicators and the generated confusion matrix, and the calculation indicators include accuracy, recall rate and F1 score.

9. The adaptive static data desensitization method based on artificial intelligence according to claim 5, It is characterized in that Deduplication of the multi-layer perceptron model according to the classification result of sensitive information includes: Use a set data structure to store data segments that the model predicts contain sensitive information; When a new data segment is predicted to contain sensitive information, it is compared with the stored data segments to obtain the similarity; If the similarity between the new data segment and the stored data segment is greater than the similarity threshold, it is considered to be a duplicate sensitive information segment and deduplication is performed; The sensitive information fragments of the combined classification results include: Determine similarity relationships based on data types; Determine whether adjacent data fragments belong to the same type of sensitive information; If adjacent data fragments are judged to belong to the same type of sensitive information, they are merged; The further screening of the classification results according to the defined rules includes: Define rules to further filter the model prediction results based on business needs and the characteristics of sensitive information; Apply rules to filter data segments predicted by the model to contain sensitive information one by one; For each data segment, check whether it satisfies all hard rules; For data segments that meet the hard rules, soft rules are further applied for screening; In the process of applying rules for screening, feedback information is continuously collected to evaluate the effectiveness and accuracy of the rules; The rules are updated and maintained regularly.

10. An adaptive static data desensitization device based on artificial intelligence, It is characterized in that include: A preprocessing module, used for acquiring static data and preprocessing the data; Feature extraction module, which is used to extract semantic and contextual features from preprocessed static data based on a bidirectional long short-term memory network model and an attention mechanism; The classification module is used to make classification decisions on static data through a multi-layer perceptron model to determine whether each data segment contains sensitive information; The post-processing module is used to optimize the multi-layer perceptron model according to the screening results of sensitive information.

Images