Deep neural network based personalized privacy risk dynamic measurement method

Abstract

The application relates to a personalized privacy risk dynamic measurement method based on a deep neural network, and specifically comprises the following steps: S1, collecting personalized privacy demands of a user, and establishing a personalized user privacy demand vector; S2, according to the personalized user privacy demand vector, extracting a feature matrix in a user picture by using a convolutional neural network, calculating a privacy feature vector from the feature matrix by using the convolutional neural network, multiplying the privacy feature vector with the personalized user privacy demand vector, obtaining a personalized privacy feature vector of the user in the picture, and identifying sensitive privacy information in the picture; and S3, predicting a leakage risk of the personalized privacy feature vector. The application is designed based on a deep neural network, solves the problem of privacy risk assessment from the perspective of user shared privacy image historical behavior characteristics, measures the risk size of user privacy information leakage, reduces the user privacy information leakage risk from the source, and enhances the user privacy consciousness.

Description

Technical Field

[0001] This invention relates to a network security technology, specifically a personalized dynamic measurement method for privacy risks based on deep neural networks. Background Technology

[0002] In recent years, online social networks have gradually become an integral part of people's lives. Currently, there are 4.2 billion social media users worldwide, and social networks have become a primary way for people to obtain and share information. On social platforms such as Facebook and WeChat, users frequently interact with each other about various aspects of life, including learning, work, and more, often involving images and media files. Analyzing these images can potentially reveal very detailed personal information about users (such as location, social relationships, and assets), which undoubtedly exposes users to risks such as spam, phishing attacks, and identity theft, causing significant losses. With the application and development of deep learning technology, more and more fields are using deep learning methods and models to analyze and model massive amounts of user data, thereby solving a series of problems in people's production and daily lives.

[0003] Social network users frequently share images on social networks, and these images may contain a large amount of private information about the users, leading to privacy leaks. User information-sharing behavior exhibits certain characteristics. Figure 1 Examples of user sharing behavior characteristics are given. In the images shared between user U0 and user U1, user U1 typically shares interesting pictures they receive with user U2, pictures about life with user U3, and pictures about work with user U4. Because everyone has different personalities and views on privacy, user U1's sharing behavior may unintentionally leak user U0's privacy information.

[0004] CN202210277708.6 discloses a method for dynamically measuring user privacy based on LSTM+CRF. This method constructs an LSTM+CRF network model to continuously extract text features from user input and compares them with privacy information in a third-party library, thereby building a measurement knowledge graph to reveal the dynamic changing trends of users' sensitive features. This measurement method measures user privacy status from the perspective of text features.

[0005] CN202210138181.9 discloses a method for measuring and protecting image privacy information. This method first redefines the image privacy information distance metric, selects a probability distribution as the proximity mechanism distribution, and generates a perturbed image from it. Then, it uses the newly defined image privacy information distance metric to measure the difference between the original image and the perturbed, protected image. Finally, the aforementioned steps are repeated until the privacy information distance metric reaches an ideal value, and the protected image is output. The privacy measurement in this method is a measure of the effectiveness of privacy information protection in privacy-preserving images.

[0006] Currently, there are few methods for measuring privacy leaks caused by image sharing on social networks. Whether an image contains sensitive private information depends on the image owner's personality traits, individual sharing preferences, level of privacy awareness, and the contextual integrity of the information content. Privacy leakage is a pre-emptive concept, and the leakage paths of private images are complex, with numerous factors influencing user sharing behavior, such as the user's personality traits and the closeness of their relationship with the target. Most existing privacy measurement methods, including the two patent documents mentioned above, calculate and quantify user privacy risks based on existing statistical information, without making users aware of the potential privacy risks associated with sharing among friends. Summary of the Invention

[0007] The purpose of this invention is to provide a personalized dynamic measurement method for privacy risks based on deep neural networks, in order to solve the problem of privacy information leakage caused by users sharing pictures in social network environments.

[0008] The objective of this invention is achieved as follows:

[0009] A personalized privacy risk dynamic measurement method based on deep neural networks includes the following steps:

[0010] S1. Collect users' personalized privacy needs and establish a personalized user privacy need vector;

[0011] S2. Based on the personalized user privacy requirement vector, use a convolutional neural network to extract the feature matrix from the user's image, calculate the privacy feature vector from the feature matrix through the convolutional neural network, and then multiply it element-wise with the personalized user privacy requirement vector to obtain the personalized privacy feature vector of the user in the image and identify the sensitive privacy information in the image.

[0012] S3. Assess and predict the risk of leakage of sensitive privacy information.

[0013] Furthermore, the specific operation method of step S1 is as follows:

[0014] S1-1. Based on existing identifiable privacy information, create a collection table to gather users' personalized privacy needs and generate a personalized privacy need vector for that user: Pri = (req1, req2, ..., req...) n ), where req i The elements in the vector represent the number of privacy feature types.

[0015] Let U = {users who believe the i-th feature contains private information}, then we have:

[0016]

[0017] If req i If (u) = 1, it means that the i-th feature belongs to the privacy features of user u; if req i If (u) = 0, it means that the information represented by the i-th feature is not the privacy information of user u.

[0018] S1-2. If user U's personalized privacy requirement vector pri contains all identifiable privacy feature elements req i If all values ​​are 0, it means that user U has no privacy requirements, and the system directly outputs a risk value of 0 to user U; otherwise, proceed with the next steps.

[0019] Furthermore, the specific operation method of step S2 is as follows:

[0020] S2-1. Input the image about user U that is about to be shared into the ResNet50 convolutional neural network to extract image features and generate feature maps;

[0021] S2-2. Feed the feature map into the region candidate network, use a sliding window to slide on the feature map in turn to generate corresponding candidate boxes, use a classifier to filter out target candidate boxes containing privacy feature regions, and obtain the corresponding feature matrix based on the projection relationship between the target candidate boxes containing privacy feature regions generated by the region candidate network and the feature map.

[0022] S2-3. Each feature matrix is ​​fed into a convolutional neural network. First, it is scaled to the same size through the RoI pooling layer. Then, it is processed by two fully connected layers (FC1, FC2) in sequence, and then processed by the classifier and the regressor to extract the privacy feature vector and determine the location of the privacy feature.

[0023] S2-4. Multiply the extracted privacy feature vector element-wise with the personalized privacy requirement vector pri of user U to obtain the personalized privacy feature vector of user U in the image.

[0024] Furthermore, the specific operation method of step S3 is as follows:

[0025] S3-1. Concatenate the sender, receiver, and sending time from each user's privacy image sharing history with the identified sensitive privacy information in the image to form an action Pol. All actions Pol form a privacy image sharing history action sequence: Seq = {Pol1, Pol2, ..., Pol...} N}; The user's i-th image sharing behavior It means user rec i Will come from user sen i Shared private images Share PR with other recipients i The privacy image dissemination record is also known as the user's historical behavior record;

[0026] S3-2. The user's privacy image sharing history sequence Seq, after embedding, is represented as: Seq'={p1,p2,…,p N}∈R N×size×dim Where size is the size of the embedding, and pi represents the embedding of the i-th row;

[0027] S3-3. The user's privacy image sharing history behavior sequence Seq after embedding is input into the GRU recurrent neural network to extract user behavior features. The calculation method is as follows:

[0028] R t =σ(p t W pr +H t-1 W hr +b r (2)

[0029] Z t =σ(p t W pz +H t-1 W hz +b z (3)

[0030] Among them, R t To reset the door, Z t For the update gate, σ(·) is the sigmoid function, p t H records users' image sharing history. t-1 W is the hidden state from the previous time step. pr W hr W pz W hz Let b be the weight matrix. r b zIt is the bias vector;

[0031] The candidate hidden state and the final hidden state at time step t are calculated as follows:

[0032]

[0033]

[0034] in, H represents the candidate hidden layer state. t For the final hidden state, ⊙ represents element-wise multiplication, and W... ph W hh Let b be the weight matrix. h It is the bias vector;

[0035] S3-4. After flattening the user's behavioral characteristics, feed them into the perceptron. Using the softmax function, calculate the probability that the receiving user may share the private image with their friends. Select the maximum value as the risk value that the sensitive private information contained in the private image may be leaked.

[0036] S3-5. Calculate the probability of leakage of multiple sensitive privacy information of the same privacy image according to formula (6):

[0037]

[0038] in, This refers to a privacy image (img) containing sensitive privacy information (k). j The risk value was leaked. This indicates the risk value of privacy image leakage, where M represents the number of friends of the recipient user, H represents the hidden layer state, and W represents the risk value of privacy image leakage. i b is a weight matrix with the corresponding shape. i It is the bias vector;

[0039] S3-6. The maximum probability of leakage of multiple sensitive privacy information from the same privacy image: The risk level of the privacy image being leaked is fed back to the sender of the privacy image.

[0040] This invention first involves the image sender defining a personalized privacy requirement vector, and then using Faster R-CNN to identify privacy attributes in the image, comprehensively constructing a sensitive privacy information vector. Next, it predicts the leakage risk of the current image based on the recipient's history of sharing private images with their friends. Finally, it comprehensively assesses the privacy leakage risk of the image. This allows for risk prediction before a user sends a private image, thus helping users reduce the risk of privacy information leakage.

[0041] This invention, based on deep neural network design, addresses privacy risk assessment by analyzing users' historical behavior patterns in sharing private images. It measures the magnitude of the risk of privacy information leakage, reducing this risk at its source and enhancing users' privacy awareness. This invention provides social network users with a quantitative and personalized method to directly assess the potential consequences of sharing, reducing the risk of privacy leaks that may result from image dissemination on social networks. The invention proposes using deep neural networks to extract behavioral features from users' image sharing history to predict personalized privacy leakage risks, minimizing the subjective influence of assessment results. Because this invention does not use users' personal configuration information, but only analyzes users' historical behavior, the difficulty of information extraction is greatly reduced, and the calculation method does not require significant modifications for different platforms, making it highly universal. Attached Figure Description

[0042] Figure 1 This is an example diagram illustrating the sharing behavior characteristics of internet users.

[0043] Figure 2 This is a flowchart of the measurement method of the present invention.

[0044] Figure 3 This is a diagram illustrating methods for identifying sensitive privacy information.

[0045] Figure 4 This is a diagram of a privacy risk prediction model. Detailed Implementation

[0046] The invention will now be described in further detail with reference to the accompanying drawings.

[0047] like Figure 2 As shown, the measurement method of this invention is generally divided into three parts: establishing a personalized user privacy demand vector, identifying sensitive privacy information in images, and predicting the risk of leakage of sensitive privacy information. Specifically, it includes the following steps:

[0048] S1. Collect users' personalized privacy needs and establish a personalized user privacy need vector. The specific operation method is as follows:

[0049] S1-1. Based on the privacy information characteristics that can be identified by the existing system, create a collection table to collect users' personalized privacy needs and generate a personalized privacy need vector for that user: Pri = (req1, req2, ..., req...) n ), where req i Let n be the element in the vector, where n represents the number of privacy feature categories that the system can recognize.

[0050] Let U = {users who believe the i-th feature contains private information}, then we have:

[0051]

[0052] When user U's privacy requirement component (element) req i When = 1, it means that user U considers the i-th feature to be a privacy feature, and the leakage of its corresponding privacy information will have a significant impact on him. When user U's privacy requirement component req i When = 0, it means that user U believes that the information represented by the i-th feature is not private information and does not care whether the private information corresponding to the feature will be leaked.

[0053] S1-2. If all elements in user U's personalized privacy requirement vector pri are 0, it means that user U has no privacy requirements, and the system directly outputs a risk value with an evaluation value of 0 to user U. Otherwise, proceed to the next step.

[0054] S2. Based on the personalized user privacy requirement vector, use a convolutional neural network to extract the feature matrix from the user's image. Calculate the privacy feature vector from the feature matrix using the convolutional neural network, and then multiply it element-wise with the personalized user privacy requirement vector to obtain the personalized privacy feature vector of the user in the image, thereby identifying sensitive privacy information in the image.

[0055] like Figure 3 As shown, this invention uses Faster R-CNN to detect whether an image contains private information during the process of identifying private images. The specific operation of step S2 is as follows:

[0056] S2-1. Input the image that user U is about to share into a ResNet50 convolutional neural network to extract the features of the image and generate a feature map.

[0057] S2-2. Feed the feature map into the region candidate network, and use a 3×3 sliding window to slide on the feature map sequentially to generate corresponding candidate boxes. Use a classifier to filter out target candidate boxes containing privacy features. Based on the projection relationship between the target candidate boxes containing privacy features generated by the region candidate network and the feature map, obtain the corresponding feature matrix.

[0058] S2-3. Each feature matrix is ​​fed into a convolutional neural network. First, it is scaled to the same size through the RoI pooling layer in the convolutional neural network. Then, it is processed by two fully connected layers (FC1, FC2), and then by the classifier and the regressor to calculate the privacy feature vector and determine the location of the privacy feature.

[0059] S2-4. Multiply the extracted privacy feature vector element-wise with the personalized privacy requirement vector pri of user U to obtain the personalized privacy feature vector of user U in the image.

[0060] Based on the personalized privacy feature vector of user U obtained from the image, the value of each element in the vector is processed differently:

[0061] If an element at a certain position is 1, then the sensitive privacy feature corresponding to that position, together with the sender's user ID and the receiver's user ID, constitutes a sensitive privacy sharing behavior: Among them, sen i This indicates the user who sent the private image, rec. i This indicates the user who received the private image. This indicates an image containing l types of sensitive privacy information that is about to be shared;

[0062] If all elements are 0, it means that the image to be shared does not contain any private information, or that user U believes that the leakage of private information contained in the image will not have any impact on him. In this case, the system will directly output a risk value of 0 to the user.

[0063] S3. Assess and predict the risk of leakage of sensitive privacy information.

[0064] An image may contain various sensitive privacy information. After identifying the sensitive privacy information in an image that a user is about to share, the various sensitive privacy information, along with the sender's user ID and the recipient's user ID, are concatenated into multiple privacy-sharing behavior records (Pol), which are then input into a privacy risk prediction model. Figure 4 This model calculates the privacy leakage risk of corresponding sensitive privacy information by analyzing the history of recipients forwarding this type of privacy information to other potential recipients. When a privacy image is shared and disseminated, it is treated as a whole; leakage of any type of sensitive privacy information within it will lead to leakage of other types of sensitive privacy information within the image. Therefore, the maximum risk of sensitive privacy information leakage within the privacy image is selected as the leakage risk value for that image. It is worth noting that although this model can predict potential recipients, it only provides the sharing probability result to the sender, because the recipient's friend relationships and sharing behavior are themselves part of the recipient's privacy.

[0065] The specific operation method for step S3 is as follows:

[0066] S3-1. Historical Behavior Records. For each user's history of sharing private images, the sender, receiver, and sending time are concatenated with the identified sensitive privacy information in the image to form a behavior Pol. All behaviors Pol form a private image sharing history behavior sequence: Seq = {Pol1, Pol2, ..., Pol...} N}, where N is the number of images shared by the user. The user's i-th image sharing action... Indicates user reci Will come from user sen i Shared private images Share PR with other recipients i The privacy image dissemination record, which is referred to as the user's historical behavior record.

[0067] S3-2. Embedding Layer. Embedding is a commonly used technique to transform high-dimensional sparse feature vectors into low-dimensional dense feature vectors. After embedding, the user ID can be represented as u. id ∈R M×dim Where M represents the number of sparse feature user IDs, and dim represents the dimension of the embedding. When a user's privacy image sharing history behavior sequence Seq is embedded, it is represented as: Seq={p1,p2,...,p...} N}∈R N×size×dim Where N is the number of user actions, size is the size of the embedding, and pi represents the embedding of the i-th action.

[0068] S3-3. User Behavior Feature Extraction. User image-sharing behavior changes with the closeness of their friendships and over time. For example, during certain periods, users may frequently share information with users they don't usually interact with, a phenomenon particularly noticeable during holidays. The GRU recurrent neural network, a variant of the LSTM recurrent neural network, not only effectively captures and learns long-term temporal relationships but also has fewer parameters than LSTM, making it faster. Therefore, this invention inputs the embedded sequence of user privacy image-sharing history (Seq) into the GRU recurrent neural network to extract user behavior features. The calculation method is as follows:

[0069] R t =σ(p t W pr +H t-1 W hr +b r (2)

[0070] Z t =σ(p t W pz +H t-1 W hz +b z (3)

[0071] Among them, R t To reset the door, Z t For the update gate, σ(·) is the sigmoid function, p tH records users' image sharing history. t-1 W is the hidden state from the previous time step. pr W hr W pz W hz Let b be the weight matrix. r b z Let be the bias vector. The candidate hidden layer states and the calculation of the hidden states at time step t are as follows:

[0072]

[0073]

[0074] in, H represents the candidate hidden layer state. t For the final hidden state, ⊙ represents element-wise multiplication, and W... ph W hh Let b be the weight matrix. h This is the bias vector.

[0075] S3-4. Output Layer. The user's behavioral features are flattened and fed into the perceptron. The softmax function is used to calculate the probability that the receiving user might share the private image with their friends. The maximum value is selected as the risk value of the sensitive private information contained in the private image being leaked.

[0076] S3-5. Calculate the probability of leakage of multiple sensitive privacy information of the same privacy image according to formula (6):

[0077]

[0078] in, This refers to a privacy image (img) containing sensitive privacy information (k). j The risk value was leaked. This indicates the risk value of privacy image leakage, where M represents the number of friends of the recipient user, H represents the hidden layer state, and W represents the risk value of privacy image leakage. i b is a weight matrix with the corresponding shape. i This is the bias vector.

[0079] S3-6. The maximum probability of leakage of multiple sensitive privacy information from the same privacy image:

[0080]

[0081] The risk value of the privacy image being leaked is fed back to the sender of the privacy image. The probability that the recipient user might share the privacy image with their friends is calculated. The privacy image may be shared with multiple friends, but once shared, the sensitive privacy information contained in the image is leaked. Therefore, the maximum value among these shared probabilities is selected as the risk value of the sensitive privacy information contained in the image being leaked. Finally, the maximum probability of leakage of various sensitive privacy information in the same privacy image is used as the risk value of the privacy image being leaked and fed back to the sender of the privacy image.

Claims

1. A personalized privacy risk dynamic measurement method based on deep neural networks, characterized in that, Includes the following steps: S1. Collect users' personalized privacy needs and establish a personalized user privacy need vector; S2. Based on the personalized user privacy requirement vector, use a convolutional neural network to extract the feature matrix from the user's image, calculate the privacy feature vector from the feature matrix through the convolutional neural network, and then multiply it element-wise with the personalized user privacy requirement vector to obtain the personalized privacy feature vector of the user in the image and identify the sensitive privacy information in the image. S3. Assess and predict the risk of leakage of sensitive privacy information; The specific operation method of step S2 is as follows: S2-1. Input the image about user U that is about to be shared into the ResNet50 convolutional neural network to extract image features and generate feature maps; S2-2. Feed the feature map into the region candidate network, use a sliding window to slide on the feature map in turn to generate corresponding candidate boxes, use a classifier to filter out target candidate boxes containing privacy feature regions, and obtain the corresponding feature matrix based on the projection relationship between the target candidate boxes containing privacy feature regions generated by the region candidate network and the feature map. S2-3. Each feature matrix is ​​fed into a convolutional neural network. First, it is scaled to the same size through the RoI pooling layer. Then, it is processed by two fully connected layers (FC1, FC2) in sequence, and then processed by the classifier and the regressor to extract the privacy feature vector and determine the location of the privacy feature. S2-4. Multiply the extracted privacy feature vector element-wise with the personalized privacy requirement vector pri of user U to obtain the personalized privacy feature vector of user U in the image.

2. The personalized privacy risk dynamic measurement method according to claim 1, characterized in that, The specific operation method of step S1 is as follows: S1-1. Based on existing identifiable privacy information, create a collection table to gather users' personalized privacy needs and generate a personalized privacy need vector for that user: Pri=(req1, req2, ..., req... n ), where req i The elements in the vector represent the number of privacy feature types. make Then we have: （1） If req i If (u)=1, it means that the i-th feature belongs to the privacy features of user u; if req i If (u)=0, it means that the information represented by the i-th feature is not the privacy information of user U; S1-2. If user U's personalized privacy requirement vector pri contains all identifiable privacy feature elements req i If all values ​​are 0, it means that user U has no privacy requirements, and the system directly outputs a risk value of 0 to user U; otherwise, proceed with the next steps.

3. The personalized privacy risk dynamic measurement method according to claim 1, characterized in that, Based on the personalized privacy feature vector of user U obtained in step S2-4, the value of each element in the vector is processed differently: If an element at a certain position is 1, then the sensitive privacy feature corresponding to that position, together with the sender's user ID and the receiver's user ID, constitutes a sensitive privacy sharing behavior: ,in, This indicates the user who sent the image. This indicates the user who received the image. Indicates that the content to be shared is included. Images containing sensitive and private information; If all elements are 0, it means that the image to be shared does not contain any private information, or that user U believes that the leakage of private information contained in the image will not have any impact on him. In this case, the system will directly output a risk value of 0 to the user.

4. The personalized privacy risk dynamic measurement method according to claim 1, characterized in that, The specific operation method of step S3 is as follows: S3-1. Concatenate the sender, receiver, and sending time from each user's history of sharing private images with the identified sensitive privacy information in the image to form an action Pol. All action Pols form a sequence of private image sharing history actions: The user's i-th image sharing behavior , represents the user Will come from users Shared private images Share with other recipients The records of the dissemination of private images are also known as user history behavior records; S3-2. The user's privacy image sharing history sequence Seq, after embedding, is represented as: Where size is the size of the embedding, p i This represents the embedding of the i-th action; S3-3. The user's privacy image sharing history behavior sequence Seq after embedding is input into the GRU recurrent neural network to extract user behavior features. The calculation method is as follows: （2） （3） in, To reset the door, To update the door, For the sigmoid function, Record users' image sharing history. This is the hidden state from the previous time step. , , , This is the weight matrix. , It is the bias vector; The candidate hidden state and the final hidden state at time step t are calculated as follows: （4） （5） in, This represents the candidate hidden layer state. This represents the final hidden state; ⊙ indicates element-wise multiplication. , This is the weight matrix. It is the bias vector; S3-4. After flattening the user's behavioral characteristics, feed them into the perceptron. Using the softmax function, calculate the probability that the receiving user may share the private image with their friends. Select the maximum value as the risk value that the sensitive private information contained in the private image may be leaked. S3-5. Calculate the probability of leakage of multiple sensitive privacy information of the same privacy image according to formula (6): （6） in, This refers to a private image containing sensitive privacy information (k). The risk value was leaked. This indicates the risk value of privacy image leakage, where M represents the number of friends of the recipient user, H represents the hidden layer state, and W represents the risk value of privacy image leakage. i b is a weight matrix with the corresponding shape. i It is the bias vector; S3-6. The maximum probability of leakage of multiple sensitive privacy information from the same privacy image: This information, representing the risk of the private image being leaked, is fed back to the sender of the private image.

Images