Private data perception model

Abstract

The invention discloses a privacy data perception model, which uses a simplified RoformerBERT model, combines an attention mechanism and RoPE coding to realize automatic perception of social network privacy information, accurately locates the position of the privacy information, uses a BI-LSTM model to carry out multiple times of coding on the information, extracts context features in a layered manner, and finally obtains the privacy data. An EGP algorithm is used to discriminate a privacy entity, the parameter utilization rate and the model identification accuracy are effectively improved, the FDPER model can accurately identify and position privacy leakage information of a user for a self-constructed microblog privacy data set and privacy data in a real social network, and the privacy data are fed back to the user, so that the privacy leakage information of the user is accurately identified and positioned, and the user experience is improved. The validity and accuracy of privacy identification are verified through the reproduction rate in specific identification privacy entity data comparative analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of privacy data security application, and in particular to a privacy data perception model. Background Art

[0002] Online social network (OSN) is a digital interactive community with multifunctional services. As an effective extension of people's real life and activities, social network fully meets people's needs to express personal demands and share various information with its convenient and flexible information dissemination and fast and efficient network communication. It has become the main carrier of opinion, information, behavior and cultural dissemination. However, most social network users have a weak awareness of personal privacy protection. The text, photos, videos, locations and other complex contents they post on social networking sites contain users' real privacy information intentionally or unintentionally. Network attackers can easily obtain a large amount of user data, and can easily mine users' privacy information through data analysis, which may lead to a series of malicious behaviors, including but not limited to tracking, defamation, spam, phishing, identity theft, personal data cloning, Sybil attacks and other privacy leakage events. Therefore, how users can effectively manage their privacy needs on social networks and how to accurately and efficiently perceive and identify privacy leakage of information posted on social networks have become urgent problems to be solved. This is not only of great theoretical significance for helping users better understand and deal with privacy risks and prevent privacy leakage, but also provides practical value for social network service providers to improve privacy protection strategies and enhance user experience.

[0003] At present, many privacy protection technologies have been proposed to address the privacy and security issues of users in social networks. The most direct approach aims to protect private information in social networks. Privacy protection technologies including anonymization, decentralization, encryption, information security supervision, fine-grained privacy settings and access control, as well as enhancing users' awareness and behavior of privacy, have been proposed. The premise of these technologies is to identify private data in social networks and perform corresponding privacy processing. This method can only identify specific categories of private information and has poor protection effect on indirect privacy leakage. At the same time, some researchers have conducted in-depth research on privacy protection in social networks based on the dynamic characteristics of social networks. They proposed to use data monitoring and data classification methods to analyze private information in dynamic social networks, and explored centralized and distributed technologies for privacy protection. Although these methods have achieved certain results in privacy protection, they cannot meet the specific needs of individuals for privacy control and pursuit of personalized protection.

[0004] The advantage of the BERT model in private entity recognition lies in its bidirectional context understanding ability and fine-grained subword processing. By pre-training on large-scale text, BERT can deeply capture contextual information, thereby accurately identifying and distinguishing various private entities. After integrating the attention mechanism, the model's ability to extract global features is enhanced. The traditional BERT model private entity recognition is not very good at identifying nested entities, and it is difficult to balance the stability and efficiency of the model. Therefore, the research plan based on the fusion of the BERT model and other models is of great significance. Summary of the invention

[0005] To achieve the above purpose, the present invention provides the following technical solutions: a privacy data perception model,

[0006] Step 1: First, preprocess the data set, split the text into words or subwords through word segmentation, then read the Vocab.txt file, convert the words in the text into corresponding vector representations according to the words and corresponding indexes, and embed the word embedding vector and the position embedding vector through RoPE re-encoding to obtain the feature expression that conforms to the input text data of the RoformerBERT pre-training model. The purpose of re-encoding with the RoPE algorithm is to convert the absolute position encoding into the relative position encoding, thereby increasing the input amount of data;

[0007] Step 2: Use the RoformerBERT model to perform feature extraction on the converted encoding vector to obtain the feature expression of the text data. After learning through the RoformerBERT model, a 1×L vector will be converted into a 348×L matrix, which can better represent the hidden features in the text. Where L represents the vector dimension of each input token;

[0008] Step 3: Input the data processed by the RoformerBERT model into the BI-LSTM model for sequence processing, thereby capturing the long-term dependencies and bidirectional information of the data and improving the performance of the model in processing sequence data. The BI-LSTM model consists of two independent LSTMs. The input sequences are in positive order w f and reverse order w b Input into two LSTM models for feature extraction, and concatenate the two output vectors to obtain the final data with sequence feature expression;

[0009] Step 4: Pass the data obtained in step 3 to the EGP algorithm for calculation. The GP layer divides the input tensor into x matrix outputs, where x is the number of entity types. By calculating each matrix, the output privacy entity category is finally determined.

[0010] Preferably, the RoformerBERT model

[0011] In order to effectively improve the recognition accuracy of nested entities and long texts, based on the BERT model, RoPE and the attention mechanism are integrated to achieve relative position encoding through absolute position encoding, which is robust to changes in position information, thereby enhancing the model's ability to extract global features. The specific steps of RoPE are as follows:

[0012] ① Add the absolute position information to q and k, where m and n represent the absolute position information, and f is an operation. After the operation f, q and k will have the absolute position information of m and n, as shown in formula (1):

[0013]

[0014] ② Using the inner product calculation and complex conjugate calculation in the attention mechanism, the inner product is converted into a form that only depends on the relative position mn. The absolute position information and relative position information are cleverly integrated, as shown in formula (2):

[0015]

[0016] Here, Re[] means that the real part of the complex number takes into account the real part of the result. imθ and e inθ Add the imaginary part to and * is conjugate calculation;

[0017] When dealing with complex tasks or needing the model to adapt to different input distributions, Layer Norm will increase computing resources and time overhead. Layer Norm is replaced with a more efficient root mean square layer normalization technology RMS Norm

[0018]

[0019] where a i is the input of the model or layer, and n is the number of samples.

[0020] Preferably, the BI-LSTM model

[0021] In order to enhance the feature extraction capability when processing tasks with long-term dependencies or very long input sequences, the BI-LSTM model is used to effectively process the temporal dependencies and long-term dependencies in the sequence through its cyclic structure, thereby improving the performance of the model in specific tasks. The BI-LSTM model consists of two LSTM networks connected to the same input layer, and the total output of the BI-LSTM model is the sum of the outputs of the two layers of LSTM.

[0022] The BI-LSTM model contains a forward LSTM layer and a backward LSTM layer.i is the hidden output sequence of the forward layer, wb i is the hidden output of the backward layer, and the overall output w of BI-LSTM i The calculation formula is shown in formula (4):

[0023]

[0024] Preferably, the EGP algorithm

[0025] The CRF algorithm currently used in most privacy entity recognition tasks shows significant advantages in sequence labeling and structure prediction. The CRF algorithm is highly complex, requires large computing resources, and performs poorly in handling nested privacy. In contrast, the GP algorithm uses the idea of ​​global normalization for privacy entity recognition. When processing multi-label sequences, it can identify nested entities and non-nested entities indiscriminately. The pseudo code of the GP algorithm principle is shown in Algorithm 1. The GP algorithm has the advantages of fast inference speed, strong generalization ability, and low demand for labeled data. It shows obvious advantages and applicability in specific multi-label sequence labeling tasks. In the case of nested NER and non-nested NER, the recognition effect of the GP algorithm is better than that of the CRF algorithm. The scoring function of the GP algorithm is shown in Formula (5):

[0026] q i,α =W q,α h i k i,α =W k,α h i

[0027] GP=Re[q i,α T k j,α e (m-n)θ ] (5)

[0028] Among them, q i,α represents the Query matrix, k i,α represents the Key matrix, h i represents the vector sequence after extracting features through BI-LSTM, W represents the weight parameter of the fully connected layer, and Re represents the relative position information injected into the scoring function through the inner product;

[0029] In order to improve parameter utilization, an EGP algorithm is proposed. The core idea is to share the score calculation for each entity type and decompose the NER task into two subtasks: extraction and classification. In the extraction step, text fragments are extracted as entities, and in the classification step, the specific type of each entity is identified. This decomposition method can equate the extraction process to a NER task containing only one entity type, which can be expressed as a score matrix (W q h i )T(Wk h j ) to complete it, the classification step can be understood as w α T [h i ;h j ], where w α ∈R 2D represents the identification of entity type α, D is the overall hidden state dimension, and the new scoring function is shown in formula (6):

[0030] s α (i,j)=(W q h i ) T (W k h j )+w α T [h i ;h j ] (6)

[0031] The parameters of the extraction task are shared by all entity types. When a new entity type is added, the parameters of the classification task increase by 2D, where D = 768. To further reduce the parameters, use [q i ;k i ]Replace h i To represent Token. The final scoring function is shown in formula (7):

[0032] s α (i,j)=q i T k j +w α T (q i ;k i ;q j ;k j ) (7)

[0033] where w α ∈R 4d ; Intuitively, the number of parameters added for each new entity type is 4d, where d is the dimension of word embedding, where d = 64, which effectively reduces the number of algorithm parameters and improves training speed.

[0034] Preferably, the loss function

[0035] Since there is too much non-entity data in the private entity recognition dataset, there is a long tail phenomenon in the dataset. The loss calculation method improves the single-target multi-classification cross entropy loss function so that the score of the private label class is higher than that of the non-private label class. For the multi-label classification task, the goal is to make the score of each target class not lower than the score of each non-target class. P is the label class, N is the non-label class, and s represents the score of each class. The loss value is calculated using the cross entropy function, as shown in the following formula (8):

[0036]

[0037] For multi-label classification with unfixed categories, a threshold is needed to determine which categories to output, so an additional 0 category is introduced. It is hoped that the scores of the target class are greater than s0, and the scores of the non-target class are less than s0, so add it to the log We get formula (9):

[0038]

[0039] When the specified threshold is 0, the final simplified Loss form is shown in the following formula (10):

[0040]

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] This paper proposes an efficient OSN privacy-aware model, FDPER, which is a combination of BERT in processing multiple applications of privacy features;

[0043] Use RoPE (Rotary Position Embedding) as position encoding, integrate the information of absolute position and relative position, and combine with the attention mechanism to effectively improve the recognition accuracy of nested entities and long texts;

[0044] As a recurrent neural network, the BI-LSTM model effectively handles temporal dependencies and long-term dependencies in sequences through its recurrent structure, which helps to identify privacy more accurately;

[0045] The EGP algorithm is introduced to use the global normalization idea for privacy entity recognition, which can identify nested entities and non-nested entities indiscriminately when processing multi-label sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the structure of the FDPER model of the present invention;

[0047] Figure 2 Schematic diagram of the BI-LSTM model of the present invention. DETAILED DESCRIPTION

[0048] See also Figure 1-2 ,The present invention provides a technical solution: a privacy data perception model,

[0049] Step 1: First, preprocess the data set, split the text into words or subwords through word segmentation, then read the Vocab.txt file, convert the words in the text into corresponding vector representations according to the words and corresponding indexes, and embed the word embedding vector and the position embedding vector through RoPE re-encoding to obtain the feature expression that conforms to the input text data of the RoformerBERT pre-training model. The purpose of re-encoding with the RoPE algorithm is to convert the absolute position encoding into the relative position encoding, thereby increasing the input amount of data;

[0050] Step 2: Use the RoformerBERT model to perform feature extraction on the converted encoding vector to obtain the feature expression of the text data. After learning through the RoformerBERT model, a 1×L vector will be converted into a 348×L matrix, which can better represent the hidden features in the text. Where L represents the vector dimension of each input token;

[0051] Step 3: Input the data processed by the RoformerBERT model into the BI-LSTM model for sequence processing, thereby capturing the long-term dependencies and bidirectional information of the data and improving the performance of the model in processing sequence data. The BI-LSTM model consists of two independent LSTMs. The input sequences are in positive order w f and reverse order w b Input into two LSTM models for feature extraction, and concatenate the two output vectors to obtain the final data with sequence feature expression;

[0052] Step 4: Pass the data obtained in step 3 to the EGP algorithm for calculation. The GP layer divides the input tensor into x matrix outputs, where x is the number of entity types. By calculating each matrix, the output privacy entity category is finally determined.

[0053] The RoformerBERT model

[0054] In order to effectively improve the recognition accuracy of nested entities and long texts, based on the BERT model, RoPE and the attention mechanism are integrated to achieve relative position encoding through absolute position encoding, which is robust to changes in position information, thereby enhancing the model's ability to extract global features. The specific steps of RoPE are as follows:

[0055] ① Add the absolute position information to q and k, where m and n represent the absolute position information, and f is an operation. After the operation f, q and k will have the absolute position information of m and n, as shown in formula (1):

[0056]

[0057] ② Using the inner product calculation and complex conjugate calculation in the attention mechanism, the inner product is converted into a form that only depends on the relative position mn. The absolute position information and relative position information are cleverly integrated, as shown in formula (2):

[0058]

[0059] Here, Re[] means that the real part of the complex number takes into account the real part of the result. imθ and e inθ Add the imaginary part to and * is conjugate calculation;

[0060] When dealing with complex tasks or needing the model to adapt to different input distributions, Layer Norm will increase computing resources and time overhead. Layer Norm is replaced with a more efficient root mean square layer normalization technology RMS Norm

[0061]

[0062] where a i is the input of the model or layer, and n is the number of samples.

[0063] The BI-LSTM model

[0064] In order to enhance the feature extraction capability when processing tasks with long-term dependencies or very long input sequences, the BI-LSTM model is used to effectively process the temporal dependencies and long-term dependencies in the sequence through its cyclic structure, thereby improving the performance of the model in specific tasks. The BI-LSTM model consists of two LSTM networks connected to the same input layer, and the total output of the BI-LSTM model is the sum of the outputs of the two layers of LSTM.

[0065] The BI-LSTM model contains a forward LSTM layer and a backward LSTM layer. i is the hidden output sequence of the forward layer, wb i is the hidden output of the backward layer, and the overall output w of BI-LSTM i The calculation formula is shown in formula (4):

[0066]

[0067] The EGP algorithm

[0068] The CRF algorithm currently used in most privacy entity recognition tasks shows significant advantages in sequence labeling and structure prediction. The CRF algorithm is highly complex, requires large computing resources, and performs poorly in handling nested privacy. In contrast, the GP algorithm uses the idea of ​​global normalization for privacy entity recognition. When processing multi-label sequences, it can identify nested entities and non-nested entities indiscriminately. The pseudo code of the GP algorithm principle is shown in Algorithm 1. The GP algorithm has the advantages of fast inference speed, strong generalization ability, and low demand for labeled data. It shows obvious advantages and applicability in specific multi-label sequence labeling tasks. In the case of nested NER and non-nested NER, the recognition effect of the GP algorithm is better than that of the CRF algorithm. The scoring function of the GP algorithm is shown in Formula (5):

[0069] q i,α =W q,α h i k i,α =W k,α h i

[0070] GP=Re[q i,α T k j,α e (m-n)θ ] (5)

[0071] Among them, q i,α represents the Query matrix, k i,α represents the Key matrix, h i represents the vector sequence after extracting features through BI-LSTM, W represents the weight parameter of the fully connected layer, and Re represents the relative position information injected into the scoring function through the inner product;

[0072] In order to improve parameter utilization, an EGP algorithm is proposed. The core idea is to share the score calculation for each entity type and decompose the NER task into two subtasks: extraction and classification. In the extraction step, text fragments are extracted as entities, and in the classification step, the specific type of each entity is identified. This decomposition method can equate the extraction process to a NER task containing only one entity type, which can be expressed as a score matrix (W q h i ) T (W k h j ) to complete it, the classification step can be understood as w α T [h i ;h j ], where w α ∈R 2Drepresents the identification of entity type α, D is the overall hidden state dimension, and the new scoring function is shown in formula (6):

[0073] s α (i,j)=(W q h i ) T (W k h j )+w α T [h i ;h j ] (6)

[0074] The parameters of the extraction task are shared by all entity types. When a new entity type is added, the parameters of the classification task increase by 2D, where D = 768. To further reduce the parameters, use [q i ;k i ]Replace h i To represent Token. The final scoring function is shown in formula (7):

[0075] s α (i,j)=q i T k j +w α T (q i ;k i ;q j ;k j ) (7)

[0076] where w α ∈R 4d ; Intuitively, the number of parameters added for each new entity type is 4d, where d is the dimension of word embedding, where d = 64, which effectively reduces the number of algorithm parameters and improves training speed.

[0077] The loss function

[0078] Since there is too much non-entity data in the private entity recognition dataset, there is a long tail phenomenon in the dataset. The loss calculation method improves the single-target multi-classification cross entropy loss function so that the score of the private label class is higher than that of the non-private label class. For the multi-label classification task, the goal is to make the score of each target class not lower than the score of each non-target class. P is the label class, N is the non-label class, and s represents the score of each class. The loss value is calculated using the cross entropy function, as shown in the following formula (8):

[0079]

[0080] For multi-label classification with unfixed categories, a threshold is needed to determine which categories to output, so an additional 0 category is introduced. It is hoped that the scores of the target class are greater than s0, and the scores of the non-target class are less than s0, so add it to the log We get formula (9):

[0081]

[0082] When the specified threshold is 0, the final simplified Loss form is shown in the following formula (10):

[0083]

[0084] The proposed FDPER (Fast Deep Learning for Recognizing Private Entities) model consists of a simplified RoformerBERT pre-trained model, a BI-LSTM model and an EGP algorithm. The simplified RoformerBERT pre-trained model can deeply learn text features and solve the imbalance problem of private entity distribution, while improving the training speed and reducing computing resources and memory. The BI-LSTM model is mainly used to extract the sequence features of sentences. The EGP algorithm effectively improves the accuracy of the model while solving the problem of private entity nesting.

Claims

1. A privacy data awareness model, characterized in that: The following steps are involved: Step 1: First, preprocess the data set, split the text into words or subwords through word segmentation, then read the Vocab.txt file, convert the words in the text into corresponding vector representations according to the words and corresponding indexes, and embed the word embedding vector and the position embedding vector through RoPE re-encoding to obtain the feature expression that conforms to the input text data of the RoformerBERT pre-training model. The purpose of re-encoding with the RoPE algorithm is to convert the absolute position encoding into the relative position encoding, thereby increasing the input amount of data; Step 2: Use the RoformerBERT model to perform feature extraction on the converted encoding vector to obtain the feature expression of the text data. After learning through the RoformerBERT model, a 1×L vector will be converted into a 348×L matrix, which can better represent the hidden features in the text. Where L represents the vector dimension of each input token; Step 3: Input the data processed by the RoformerBERT model into the BI-LSTM model for sequence processing, thereby capturing the long-term dependencies and bidirectional information of the data and improving the performance of the model in processing sequence data. The BI-LSTM model consists of two independent LSTMs. The input sequences are in positive order w f and reverse order w b Input into two LSTM models for feature extraction, and concatenate the two output vectors to obtain the final data with sequence feature expression; Step 4: Pass the data obtained in step 3 to the EGP algorithm for calculation. The GP layer divides the input tensor into x matrix outputs, where x is the number of entity types. By calculating each matrix, the output privacy entity category is finally determined.

2. A privacy data awareness model according to claim 1, characterized in that: The RoformerBERT model In order to effectively improve the recognition accuracy of nested entities and long texts, based on the BERT model, RoPE and the attention mechanism are integrated to achieve relative position encoding through absolute position encoding, which is robust to changes in position information, thereby enhancing the model's ability to extract global features. The specific steps of RoPE are as follows: ① Add the absolute position information to q and k, where m and n represent the absolute position information, and f is an operation. After the operation f, q and k will have the absolute position information of m and n, as shown in formula (1): ② Using the inner product calculation and complex conjugate calculation in the attention mechanism, the inner product is converted into a form that only depends on the relative position mn. The absolute position information and relative position information are cleverly integrated, as shown in formula (2): Here, Re[ ] means that the real part of the complex number takes into account the real part of the result. imθ and e inθ Add the imaginary part to and * is conjugate calculation; When dealing with complex tasks or needing the model to adapt to different input distributions, Layer Norm will increase computing resources and time overhead. Layer Norm is replaced with a more efficient root mean square layer normalization technology RMS Norm where a i is the input of the model or layer, and n is the number of samples.

3. A privacy data awareness model according to claim 1, characterized in that: The BI-LSTM model In order to enhance the feature extraction capability when processing tasks with long-term dependencies or very long input sequences, the BI-LSTM model is used to effectively process the temporal dependencies and long-term dependencies in the sequence through its cyclic structure, thereby improving the performance of the model in specific tasks. The BI-LSTM model consists of two LSTM networks connected to the same input layer, and the total output of the BI-LSTM model is the sum of the outputs of the two layers of LSTM. The BI-LSTM model contains a forward LSTM layer and a backward LSTM layer. i is the hidden output sequence of the forward layer, wb i is the hidden output of the backward layer, and the overall output w of BI-LSTM i The calculation formula is shown in formula (4):

4. The privacy data awareness model according to claim 1, characterized in that: The EGP algorithm The CRF algorithm currently used in most privacy entity recognition tasks shows significant advantages in sequence labeling and structure prediction. The CRF algorithm is highly complex, requires large computing resources, and performs poorly in handling nested privacy. In contrast, the GP algorithm uses the idea of ​​global normalization for private entity recognition. When processing multi-label sequences, it can identify nested entities and non-nested entities indiscriminately. The pseudo code of the GP algorithm principle is shown in Algorithm 1. The GP algorithm has the advantages of fast inference speed, strong generalization ability and less demand for labeled data. It shows obvious advantages and applicability in specific multi-label sequence labeling tasks. In the case of nested NER and non-nested NER, the recognition effect of the GP algorithm is better than that of the CRF algorithm. The scoring function of the GP algorithm is shown in Formula (5): q i,α =W q,α h i k i,α =W k,α h i GP=Re[q i,α T k j,α e (m-n)θ ] (5) Among them, q i,α represents the Query matrix, k i,α represents the Key matrix, h i represents the vector sequence after extracting features through BI-LSTM, W represents the weight parameter of the fully connected layer, and Re represents the relative position information injected into the scoring function through the inner product; In order to improve parameter utilization, an EGP algorithm is proposed. The core idea is to share the score calculation for each entity type and decompose the NER task into two subtasks: extraction and classification. In the extraction step, text fragments are extracted as entities, and in the classification step, the specific type of each entity is identified. This decomposition method can equate the extraction process to a NER task containing only one entity type, which can be expressed as a score matrix (W q h i ) T (W k h j ) to complete it, the classification step can be understood as w α T [h i ;h j ], where w α ∈R 2D represents the identification of entity type α, D is the overall hidden state dimension, and the new scoring function is shown in formula (6): s α (i,j)=(W q h i ) T (W k h j )+w α T [h i ;h j ] (6) The parameters of the extraction task are shared by all entity types. When a new entity type is added, the parameters of the classification task increase by 2D, where D = 768. To further reduce the parameters, use [q i ;k i ]Replace h i To represent Token. The final scoring function is shown in formula (7): s α (i,j)=q i T k j +w α T (q i ;k i ;q j ;k j ) (7) where w α ∈R 4d ; Intuitively, the number of parameters added for each new entity type is 4d, where d is the dimension of word embedding, where d = 64, which effectively reduces the number of algorithm parameters and improves training speed.

5. The privacy data awareness model according to claim 1, characterized in that: The loss function Since there is too much non-entity data in the private entity recognition dataset, there is a long tail phenomenon in the dataset. The loss calculation method improves the single-target multi-classification cross entropy loss function so that the score of the private label class is higher than that of the non-private label class. For the multi-label classification task, the goal is to make the score of each target class not lower than the score of each non-target class. P is the label class, N is the non-label class, and s represents the score of each class. The loss value is calculated using the cross entropy function, as shown in the following formula (8): For multi-label classification with unfixed categories, a threshold is needed to determine which categories to output, so an additional 0 category is introduced. It is hoped that the scores of the target class are greater than s0, and the scores of the non-target class are less than s0, so add it to the log We get formula (9): When the specified threshold is 0, the final simplified Loss form is shown in the following formula (10):

Images