A method for constructing a privacy data perception model

Abstract

The present invention discloses a method for constructing a privacy data perception model. By using a simplified RoformerBERT model, combining the attention mechanism and RoPE encoding to achieve the automatic perception of social network privacy information, accurately locate the position of privacy information, and using the BI-LSTM model to encode the text information multiple times, extract context features hierarchically, and then use the EGP algorithm to discriminate privacy entities, effectively improving the parameter utilization rate and the recognition accuracy of the model. For the self-constructed Weibo privacy dataset and privacy data in the real social network, the FDPER model can accurately identify and locate the user's privacy leakage information and feedback this privacy data to the user. In the specific comparative analysis of identifying privacy entity data, the effectiveness and accuracy of privacy recognition are also confirmed by the reproduction rate.

Description

Technical Field

[0001] The present invention relates to the technical field of privacy data security applications, and specifically to a method for constructing a privacy data perception model. Background Art

[0002] An online social network (OSN) is a digital interaction community with multifunctional services. As an effective extension of people's real life and activities, the social network fully meets people's needs for expressing personal appeals and sharing various information with its convenient, flexible information dissemination and fast and efficient network communication, and has become the main carrier for the dissemination of opinions, information, behaviors, and culture. However, most social network users have relatively weak awareness of personal privacy protection. The various forms of complex content such as text, photos, videos, and location information they post on social websites may inadvertently contain users' real privacy information. Network attackers can easily obtain a large amount of user data and can easily mine users' privacy information through data analysis, which may lead to a series of malicious behaviors, including but not limited to privacy leakage incidents such as stalking, slander, spam, phishing, identity theft, personal data cloning, and Sybil attacks. Therefore, how users can effectively manage their privacy needs on social networks, and how to accurately and efficiently perceive and identify the privacy leakage of information posted on social networks has become an urgent problem to be solved. This not only has important theoretical significance for helping users better understand and cope with privacy risks and prevent privacy leakage, but also provides practical value for social network service providers to improve privacy protection strategies and enhance user experience.

[0003] Currently, many privacy protection technologies have been proposed for the privacy and security issues of users in social networks. The most direct method aims to protect privacy information in social networks. Privacy protection technologies including anonymization, decentralization, encryption, information security supervision, fine-grained privacy settings, and access control have been proposed, as well as enhancing users' awareness and behaviors regarding privacy. The premise of these technologies is to identify privacy data in social networks and perform corresponding privacy processing. This method can only identify specific categories of privacy information and has poor protection effect for indirect privacy leakage. At the same time, some researchers have conducted in-depth research on privacy protection in social networks based on the dynamic characteristics of social networks. They proposed using data monitoring and data classification methods to analyze privacy information in dynamic social networks and explored centralized and distributed technologies for privacy protection. Although these methods have achieved certain results in privacy protection, they cannot meet the specific needs of individuals for privacy control and the pursuit of personalized protection.

[0004] The advantages of the BERT model in privacy entity recognition lie in its bidirectional context understanding ability and fine-grained sub-word processing. By pre-training on a large-scale text, BERT can deeply capture context information, thereby accurately identifying and distinguishing various privacy entities. After integrating the attention mechanism, it enhances the model's ability to extract global features. Traditional BERT model-based privacy entity recognition is not very good at recognizing nested entities, and it is difficult to balance the stability and efficiency of the model. Therefore, research solutions based on the fusion of the BERT model and other models are of great significance. Summary of the Invention

[0005] To achieve the above object, the present invention provides the following technical solution: A method for constructing a privacy data-aware model,

[0006] Step 1: First, perform preprocessing operations on the data set. The text is segmented into words or sub-words through word segmentation operations. Then, read the Vocab.txt file, and convert the words in the text into corresponding vector representations according to the words and their corresponding indices. The word embedding vector and the position embedding vector are fused through RoPE re-encoding embedding to obtain a feature expression that conforms to the input text data of the RoformerBERT pre-trained model. The purpose of using the RoPE algorithm for re-encoding is to convert the absolute position encoding into relative position encoding, thereby increasing the input volume of the data;

[0007] Step 2: Use the RoformerBERT model to perform feature extraction processing on the converted encoded vector to obtain a feature expression of the text data. After learning through the RoformerBERT model, a vector will be converted into a matrix, which can better represent the hidden features in the text. Among them, represents the vector dimension of each input token;

[0008] Step 3: Input the data processed by the RoformerBERT model into the BI-LSTM model for sequence processing, thereby capturing the long-term dependence relationship and bidirectional information of the data, and improving the performance of the model when processing sequence data. The BI-LSTM model consists of two independent LSTMs. The input sequence is input into the two LSTM models for feature extraction in the forward and reverse orders respectively, and the two output vectors are concatenated to obtain the final data with sequence feature expression;

[0009] Step 4: Transmit the data obtained in Step 3 to the EGP algorithm for calculation. The GP layer divides the input tensor into matrix outputs, where is the number of entity types. By calculating each matrix, the output privacy entity category is finally determined.

[0010] Preferably, the RoformerBERT model

[0011] To effectively improve the recognition accuracy of nested entities and long texts, we integrate RoPE with the attention mechanism on the basis of the BERT model. We implement relative position encoding through absolute position encoding, which is robust to changes in position information and enhances the model's ability to extract global features. The specific steps of RoPE are as follows:

[0012] ①Add absolute position information to and Among them and Indicates absolute position information, is an operation, After operation and will have and The absolute position information is as shown in formula (1):

[0013]

[0014] ②Use the inner product calculation and complex conjugate calculation in the attention mechanism to transform the inner product into a function that only depends on the relative position The absolute position information and relative position information are cleverly integrated, as shown in formula (2):

[0015]

[0016] in, Representing the real part of a complex number takes into account the real part of the result. and Indicates adding the imaginary part to and , For conjugate calculation;

[0017] When dealing with complex tasks or requiring the model to adapt to different input distributions, Layer Norm will increase computing resources and time overhead. Layer Norm is replaced with the more efficient root mean square layer normalization technology RMS Norm

[0018]

[0019] in is the input to the model or layer, is the sample size.

[0020] Preferably, the BI-LSTM model

[0021] When dealing with tasks of long-term dependencies or very long input sequences to enhance its feature extraction ability, the BI-LSTM model effectively processes the temporal dependencies and long-term dependencies in the sequence through its recurrent structure, thereby improving the performance of the model in specific tasks; the BI-LSTM model consists of forward and backward LSTM networks connected to the same input layer, and the total output of the BI-LSTM model is the sum of the outputs of the two LSTM layers;

[0022] The BI-LSTM model includes a forward LSTM layer and a backward LSTM layer. is the hidden output sequence of the forward layer, is the hidden output of the backward layer, and the overall output of the BI-LSTM The calculation formula of is shown in formula (4):

[0023]

[0024] Preferably, the EGP algorithm

[0025] The CRF algorithm used in most current privacy entity recognition tasks shows significant advantages in sequence labeling and structure prediction. The CRF algorithm has high complexity, large computational resource requirements, and poor performance in dealing with nested privacy; in contrast, the GP algorithm uses the idea of global normalization for privacy entity recognition and can identify nested entities and non-nested entities without discrimination when dealing with multi-label sequences. The principle pseudo-code of the GP algorithm is shown in Algorithm 1. The GP algorithm has the advantages of fast inference speed, strong generalization ability, and less demand for labeled data, and shows obvious advantages and applicability in specific multi-label sequence labeling tasks. In the case of nested NER and non-nested NER, the recognition effect of the GP algorithm is better than that of the CRF algorithm. The scoring function of the GP algorithm is shown in formula (5):

[0026]

[0027] Among them, represents the Query matrix, represents the Key matrix, represents the vector sequence after extracting features through the BI-LSTM, represents the weight parameter of the fully connected layer, represents injecting relative position information into the scoring function through the inner product;

[0028] To improve the parameter utilization rate, an EGP algorithm is proposed. The core idea is to share the score calculation for each entity type, decompose the NER task into two subtasks: extraction and classification. In the extraction step, text fragments are extracted as entities, and in the classification step, the specific type of each entity is identified. This decomposition method can make the extraction process equivalent to an NER task that only contains one entity type, and it can be completed with a scoring matrix The classification step can be understood as , where represents the identification of the entity type , is the overall hidden state dimension. The new scoring function is shown in formula (6):

[0029]

[0030] The parameters of the extraction task are shared by all entity types. When a new entity type is added, the parameters of the classification task increase , where ; To further reduce the parameters, is used to replace to represent the Token. The final scoring function is shown in formula (7):

[0031]

[0032] where ; Intuitively, the number of parameters increased by each new entity type is , is the dimension of the word embedding, where , effectively reducing the number of algorithm parameters and improving the training speed.

[0033] Preferably, the loss function

[0034] Due to the excessive non-entity data in the private entity recognition dataset, there is a long-tail phenomenon in the dataset. The loss calculation method improves the single-objective multi-class cross-entropy loss function, making the score of the private label class higher than that of the non-private label class. For the multi-label classification task, the goal is to make the score of each target category not lower than that of each non-target category. is the label class, is the non-label class, represents the score of each class; The loss value is calculated using the cross-entropy function, as shown in the following formula (8):

[0035]

[0036] For multi-label classification with unfixed categories, a threshold is needed to determine which classes to output. Therefore, an additional class 0 is introduced, hoping that the scores of the target classes are all greater than , the scores of non-target classes are all less than , so add into it to obtain formula (9):

[0037]

[0038] When the specified threshold is 0, the finally simplified Loss form is as shown in formula (10):

[0039]

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] The present invention proposes an efficient OSN privacy-aware model FDPER, which is a combination of BERT in various applications for processing privacy features;

[0042] Use RoPE (Rotary Position Embedding) as the position encoding, fuse the information of absolute position and relative position, and effectively improve the recognition accuracy of nested entities and long texts in combination with the attention mechanism;

[0043] The BI-LSTM model, as a recurrent neural network, effectively processes the temporal dependence and long-term dependence relationships in the sequence through its recurrent structure, which helps to identify privacy more accurately;

[0044] Introduce the EGP algorithm to perform privacy entity recognition using the idea of global normalization, and can identify nested entities and non-nested entities without discrimination when processing multi-label sequences. Brief Description of the Drawings

[0045] Figure 1 It is a schematic diagram of the FDPER model structure of the present invention;

[0046] Figure 2 It is a schematic diagram of the BI-LSTM model of the present invention. Detailed Embodiments

[0047] Please refer to Figure 1-2 , the present invention provides a technical solution: a method for constructing a privacy data perception model,

[0048] Step 1: First, preprocess the dataset. Split the text into words or sub-words through tokenization. Then, read the Vocab.txt file and convert the words in the text into corresponding vector representations according to the words and their corresponding indices. Fuse the word embedding vectors and position embedding vectors through RoPE re-encoding embedding to obtain a feature expression that conforms to the input text data of the RoformerBERT pre-trained model. The purpose of using the RoPE algorithm for re-encoding is to convert the absolute position encoding into relative position encoding, thereby increasing the amount of input data;

[0049] Step 2: Use the RoformerBERT model to perform feature extraction on the converted encoded vectors to obtain a feature expression of the text data. After learning through the RoformerBERT model, a vector will be transformed into a matrix, which can better represent the hidden features in the text. Among them, represents the vector dimension of each input token;

[0050] Step 3: Input the data processed by the RoformerBERT model into the BI-LSTM model for sequence processing, thereby capturing the long-term dependencies and bidirectional information of the data, and improving the performance of the model when processing sequence data. The BI-LSTM model consists of two independent LSTMs. The input sequence is input into the two LSTM models for feature extraction in forward and reverse orders respectively, and the two output vectors are concatenated to obtain the final data with sequence feature expressions;

[0051] Step 4: Pass the data obtained in Step 3 into the EGP algorithm for calculation. The GP layer divides the input tensor into matrix outputs, where is the number of entity types. By calculating each matrix, the privacy entity category of the output is finally determined.

[0052] The RoformerBERT model

[0053] To effectively improve the recognition accuracy of nested entities and long texts, on the basis of the BERT model, fuse RoPE and the attention mechanism, achieve relative position encoding through absolute position encoding, and be robust to changes in position information, thereby enhancing the model's ability to extract global features. The specific steps of RoPE are as follows:

[0054] ① Add the absolute position information to and where and represent the absolute position information, is an operation that, after the operation, and will have and absolute position information as shown in formula (1):

[0055]

[0056] ② By using the inner product calculation and complex conjugate calculation in the attention mechanism, the inner product is transformed into a form that only depends on the relative position. The absolute position information and relative position information are cleverly fused as shown in formula (2):

[0057]

[0058] where represents the real part of the complex number, considering the real part of the result. and represent adding the imaginary part to and , is the conjugate calculation;

[0059] In the case of dealing with complex tasks or when the model needs to adapt to different input distributions, Layer Norm increases the computational resources and time overhead. RMS Norm, a more efficient root mean square layer normalization technique, is used instead of Layer Norm.

[0060]

[0061] where is the input of the model or layer, is the number of samples.

[0062] The BI-LSTM model

[0063] To handle tasks with long-term dependencies or very long input sequences and enhance its feature extraction ability, the BI-LSTM model effectively processes the temporal dependencies and long-term dependencies in the sequence through its recurrent structure, thereby improving the performance of the model in specific tasks; the BI-LSTM model consists of forward and backward LSTM networks connected to the same input layer, and the total output of the BI-LSTM model is the sum of the outputs of the two LSTM layers;

[0064] The BI-LSTM model includes a forward LSTM layer and a backward LSTM layer. is the hidden output sequence of the forward layer, is the hidden output of the backward layer, and the overall output of the BI-LSTM The calculation formula is as shown in formula (4):

[0065]

[0066] The EGP algorithm

[0067] In most current privacy entity recognition tasks, the CRF algorithm used shows significant advantages in sequence labeling and structure prediction. However, the CRF algorithm has high complexity, large computational resource requirements, and poor performance in dealing with nested privacy. In contrast, the GP algorithm uses the idea of global normalization for privacy entity recognition and can identify nested entities and non-nested entities without discrimination when dealing with multi-label sequences. The principle pseudocode of the GP algorithm is shown in Algorithm 1. The GP algorithm has the advantages of fast inference speed, strong generalization ability, and less need for labeled data, and shows obvious advantages and applicability in specific multi-label sequence labeling tasks. In the case of nested NER and non-nested NER, the recognition effect of the GP algorithm is better than that of the CRF algorithm. The scoring function of the GP algorithm is as shown in formula (5):

[0068]

[0069] Among them, represents the Query matrix, represents the Key matrix, represents the vector sequence after extracting features through BI-LSTM, represents the weight parameter of the fully connected layer, represents injecting relative position information into the scoring function through the inner product;

[0070] To improve the parameter utilization rate, an EGP algorithm is proposed. Its core idea is to share the score calculation for each entity type, decompose the NER task into two subtasks: extraction and classification. In the extraction step, text fragments are extracted as entities, and in the classification step, the specific type of each entity is identified. This decomposition method can make the extraction process equivalent to a NER task containing only one entity type, which can be completed with the scoring matrix The classification step can be understood as where represents the identification of the entity type and is the overall hidden state dimension. The new scoring function is as shown in formula (6):

[0071]

[0072] The parameters of the extraction task are shared by all entity types. When a new entity type is added, the parameters of the classification task increase where ; To further reduce the parameters, use Replace to represent a Token. The final scoring function is shown in formula (7):

[0073]

[0074] where ; Intuitively, the number of parameters added for each new entity type is , is the dimension of the word embedding, where , effectively reducing the number of algorithm parameters and improving the training speed.

[0075] The loss function

[0076] Since there is too much non-entity data in the private entity recognition dataset, resulting in a long-tail phenomenon in the dataset, the loss calculation method improves the single-objective multi-class cross-entropy loss function, making the scores of private label classes higher than those of non-private label classes. For multi-label classification tasks, the goal is to make the scores of each target class not lower than those of each non-target class. is the label class,[[]] is the non-label class,[[]] represents the score of each class; The loss value is calculated using the cross-entropy function, as shown in the following formula (8):

[0077]

[0078] For multi-label classification with an unfixed number of classes, a threshold is needed to determine which classes to output. Therefore, an additional class 0 is introduced, hoping that the scores of target classes are all greater than , and the scores of non-target classes are all less than , so add into to get formula (9):

[0079]

[0080] When the specified threshold is 0, the final simplified Loss form is shown in the following formula (10):

[0081]

[0082] The proposed FDPER (Fast Deep Learning for Recognizing Private Entities) model consists of a simplified RoformerBERT pre-trained model, a BI-LSTM model, and an EGP algorithm. The simplified RoformerBERT pre-trained model can deeply learn text features and address the imbalance in the distribution of private entities, reducing computational resources and memory while improving the training speed. The BI-LSTM model is mainly used to extract the sequential features of sentences, and the EGP algorithm effectively improves the accuracy of the model while solving the problem of private entity nesting.

Claims

1. A method for constructing a privacy data perception model, characterized in that, It includes the following steps: Step 1: First, preprocess the dataset. Split the text into words or sub-words through tokenization. Then read the Vocab.txt file and convert the words in the text into corresponding vector representations according to the words and their corresponding indices. Fuse the word embedding vectors and position embedding vectors through RoPE re-encoding embedding to obtain a feature expression that conforms to the input text data of the RoformerBERT pre-trained model. The purpose of using the RoPE algorithm for re-encoding is to convert the absolute position encoding into relative position encoding, thereby increasing the amount of data input; Step 2: Use the RoformerBERT model to perform feature extraction processing on the transformed encoded vectors to obtain the feature expression of the text data. After learning by the RoformerBERT model, a vector will be transformed into a matrix, which can better represent the hidden features in the text; wherein represents the vector dimension of each input token; Step 3: Input the data processed by the RoformerBERT model into the BI-LSTM model for sequence processing, thereby capturing the long-term dependencies and bidirectional information of the data, improving the performance of the model in processing sequence data. The BI-LSTM model consists of two independent LSTMs. The input sequence is input into the two LSTM models for feature extraction in the forward order and reverse order respectively. The two output vectors are concatenated to obtain the final data with sequence feature representation; Step 4: Pass the data obtained in Step 3 into the EGP algorithm for calculation. The GP layer divides the input tensor into matrix outputs, where is the number of entity types. By calculating each matrix, the privacy entity category of the output is finally determined; The core idea of the EGP algorithm is to calculate the shared score for each entity type, decompose the NER task into two subtasks: extraction and classification. In the extraction step, text fragments are extracted as entities, and in the classification step, the specific type of each entity is identified. This decomposition method can make the extraction process equivalent to the NER task containing only one entity type, which can be completed with a scoring matrix The classification step can be understood as , where represents the identification of the entity type , is the overall hidden state dimension, and the new scoring function is shown in formula (6): ； The parameters of the extraction task are shared by all entity types, and when a new entity type is added, the parameters of the classification task increase , where ; To further reduce the parameters, use to replace to represent the Token, and the final scoring function is shown in formula (7): ； Among them ; Intuitively, the number of parameters added for each new entity type is , is the dimension of the word embedding, where .

2. The method for constructing a privacy data perception model according to claim 1, wherein: For the RoformerBERT model, in order to effectively improve the recognition accuracy of nested entities and long texts, on the basis of the BERT model, fuse RoPE and the attention mechanism, achieve relative position encoding through absolute position encoding, and be robust to changes in position information, thereby enhancing the model's ability to extract global features. The specific steps of RoPE are as follows: Add absolute position information to and where and represent absolute position information, is an operation that, after the operation in will have and the absolute position information of and as shown in formula (1): ; ② By using the inner product calculation and complex conjugate calculation in the attention mechanism, the inner product is transformed into a form that only depends on the relative position , and the absolute position information and relative position information are cleverly fused, as shown in formula (2): ； Among them, represents the real part of the complex number considering the real part of the result, and represents adding the imaginary part to and , for conjugate calculation; In the case of dealing with complex tasks or when the model needs to adapt to different input distributions, Layer Norm increases the computational resources and time overhead, and Layer Norm is replaced with the more efficient root mean square layer normalization technique RMS Norm ； wherein is the input of the model or layer, is the number of samples.

3. A method for constructing a privacy data perception model according to claim 1, characterized in that: For the BI-LSTM model, in order to handle tasks with long-term dependencies or very long input sequences and enhance its feature extraction ability, use the BI-LSTM model to effectively process the temporal dependence and long-term dependence relationships in the sequence through its cyclic structure, thereby improving the model's performance in specific tasks; The BI-LSTM model consists of forward and backward LSTM networks connected to the same input layer, and the total output of the BI-LSTM model is the sum of the outputs of the two layers of LSTM; The BI-LSTM model contains a forward LSTM layer and a backward LSTM layer. is the hidden output sequence of the forward layer. is the hidden output of the backward layer, and the overall output of the BI-LSTM is calculated as shown in Equation (4): 。 4. A method for constructing a privacy data perception model according to claim 1, characterized in that: For the loss function, due to the excessive amount of non-entity data in the private entity recognition dataset, there is a long-tail phenomenon in the dataset. The loss calculation method improves the single-objective multi-class cross-entropy loss function to make the score of the private label class higher than that of the non-private label class. For the multi-label classification task, the goal is to make the score of each target category not lower than that of each non-target category. is the label class, is the non-label class, represents the score of each class; the loss value is calculated using the cross-entropy function, as shown in the following formula (8): ； For multi-label classification with an unfixed number of classes, a threshold is needed to determine which classes to output. Therefore, an additional class 0 is introduced, hoping that the scores of the target classes are all greater than , and the scores of the non-target classes are all less than . So, add to to obtain Equation (9): ； When the specified threshold is 0, the final simplified Loss form is shown in formula (10) as follows: 。

Images