A generalized scene graph based multi-modal privacy leakage analysis framework

Abstract

The application discloses a multi-modal privacy leakage analysis method based on a generalized scene graph, analyzes privacy leakage caused by correlation of different modal information published on different platforms, automatically senses key features in information by using an attention mechanism, models correlation between the features and privacy attributes, reveals a principle of privacy leakage caused by multi-modal data correlation analysis, and establishes a complete data publishing privacy leakage analysis system. Scene graph feature extraction modules are respectively output for feature extraction on image and text data, multi-modal data features are fused in an MLP (Multilayer Perceptron) layer to construct a generalized scene graph, features in the generalized scene graph are adjusted in weight through a Transformer module, a co-occurrence relationship of features corresponding to privacy attributes is learned, and finally, a possibility of privacy leakage is output.

Description

Technical Field

[0001] This invention belongs to the field of privacy protection and is a multimodal privacy leakage analysis framework based on generalized scene graphs. Background Technology

[0002] The development of the internet has led to the generation of massive amounts of data, with various types of data being updated daily, including images, text, video, audio, and structured data. When enough data accumulates, it can be analyzed to reveal patterns hidden behind the data. Data closely related to people, such as chat messages and personal updates in social media, shopping records and personal information in shopping apps, and search history and location information in lifestyle apps, can be analyzed to infer certain aspects of a person's privacy.

[0003] Although privacy and security have gained increasing attention from individuals and organizations in recent years, and various privacy protection methods are employed to safeguard data during publication, preventing the discovery of valuable information even if the data is obtained, the risk of privacy breaches still exists when different data are correlated. While the risk of privacy breaches during multi-source, multimodal data correlation analysis is widely acknowledged, related work is currently limited. However, correlation privacy analysis is prevalent in real life, often difficult to detect, and carries significant risks; cyberbullying is a concrete example. Research on correlation privacy can, on the one hand, further enhance our understanding of privacy and explore correlation privacy protection issues, and on the other hand, provide a more comprehensive assessment of privacy breach risks. Privacy can be viewed as an attribute that an individual is unwilling to disclose. This attribute manifests in data and is related to certain characteristics, which may differ or be incomplete across different media. Through correlation analysis of large amounts of multi-source, multimodal data, the relationship between hidden attributes and characteristics can be established, thereby enabling privacy analysis and potentially leading to privacy breaches. Summary of the Invention

[0004] To address the current lack of privacy leakage prevention measures when using association analysis with multi-source, multi-modal data, this paper designs a multi-source, multi-modal association privacy analysis framework to solve the problem of ignoring potentially leaked privacy information when publishing multi-modal data.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A multimodal privacy leakage analysis framework based on generalized scene graphs includes the following steps:

[0007] Step 1: Input the text and image into the corresponding scene graph feature extraction module for feature extraction, forming feature triples;

[0008] Step 2: Enrich the overall feature information of the text by passing it through a self-attention layer, and then align the scene graph features of the image with the overall feature information through a Multilayer Perceptron layer to construct a generalized scene graph.

[0009] Step 3: Input the generalized scene graph into the Transformer to model the relationship between features and privacy attributes and output the results.

[0010] Step 4: Calculate the loss function for the output and the true labels. Repeat the above steps until the loss function converges.

[0011] Furthermore, step one includes the following steps:

[0012] The input text data is plain English text, and the images are in JPG and JPEG formats. The image feature extraction module uses an attention-based scene graph generation method. First, a ResNet50 network extracts image features, which are then processed by a feature decoder and an entity decoder to obtain subject-object pairs. These pairs are then decoded into triples using a feedforward network layer and an attention layer. The text is then processed by a pre-trained text scene graph generation model, which extracts the text data into triples.

[0013] Furthermore, step two includes the following steps:

[0014] It executes the rich overall features of the text's inherent information, performs unified representation of cross-modal data, and integrates features of multimodal data.

[0015]

[0016] Where: Attention represents the text self-attention layer. LN represents the connection operation, LN represents the MLP layer, and σ represents the activation function. This indicates the extraction of overall text features. G represents the rich overall characteristics of the text's inherent information. Ti Represents a text scene diagram. and G represents the text and image scene diagrams after dimension adjustment, respectively. i This represents the constructed generalized scene graph.

[0017] The text self-attention layer has a sentence input size of 1×1×300, a triple input size of 1×3×300, and an output size of 1×1×300. The text scene graph has a dimension of 1×(T+1)×300, where T is the number of triples in a sentence, thus obtaining text features. The image scene graph has a dimension of 1×I×H×W, where I is the number of subject-object pairs in the image, and H and W are the height and width of the image feature map, respectively. The LN is an MLP consisting of two linear layers and two activation layers. The text linear layer has an input size of 300, a hidden layer dimension of 1024, and an output dimension of 2048. The image linear layer has an input dimension of H×W, a hidden layer dimension of 1024, and an output dimension of 2048.

[0018] Furthermore, step three includes the following steps:

[0019] The Transformer has 18 layers. The input to the Encoder layer is a text scene graph, which is then fused with intrinsic text information through an 8-head self-attention layer and a feedforward network to establish autocorrelation associations of text features. The input to the Decoder layer is the output of the Encoder layer and an image scene graph. The image scene graph is fused with intrinsic image information through an 8-head self-attention layer, and then combined with the output of the Encoder layer through an 8-head attention layer to associate the two modalities, performing cross-modal data fusion. Finally, the output is passed through a feedforward network, and privacy prediction is performed based on this result. Figure 3 The structure diagram is shown below.

[0020] Furthermore, step four includes the following steps:

[0021] The feedforward network output obtained in step three is mapped to the privacy label dimension through a global average pooling layer and a linear layer, resulting in the final output. Dimensions greater than 0.7 in the result are set to 1, and the loss function is calculated using the ground truth labels. This process is repeated until the loss function converges.

[0022] Compared with the prior art, the present invention has the following innovations:

[0023] This invention addresses the privacy leakage problem that may result from multimodal data correlation analysis by proposing a multimodal privacy leakage analysis framework based on generalized scene graphs. This framework enables the detection of the possibility of privacy leakage in end-to-end multimodal data and provides a new way to assess the privacy leakage risks brought about by big data environments. Attached Figure Description

[0024] Figure 1 This is a flowchart of the present invention;

[0025] Figure 2 It is the attention module of the text scene graph in the framework of this invention;

[0026] Figure 3 This is the overall framework diagram of the present invention. Detailed Implementation

[0027] The invention will be further illustrated below with reference to examples.

[0028] This invention provides a multimodal privacy leakage analysis framework based on a generalized scene graph, including feature extraction, generalized scene graph generation, modeling the correlation between features and privacy attributes, and finally outputting privacy leakage results. (The following is a summary...) Figure 1-3 The following examples further illustrate this point.

[0029] like Figure 1 As shown, a multimodal privacy leakage analysis framework based on a generalized scene graph includes the following steps:

[0030] Step 1: The input text data is plain English text, and the images are in JPG and JPEG formats. The image feature extraction module uses a scene graph generation method based on an attention mechanism. First, a ResNet50 network extracts image features, which are then processed by a feature decoder and an entity decoder to obtain subject-object pairs. These pairs are then decoded into triples using a feedforward network layer and an attention layer. The text is then processed by a pre-trained text scene graph generation model, which extracts the text data into triples.

[0031] Step Two, as follows Figure 2 As shown, the text's inherent information enriches the overall features, performs unified representation of cross-modal data, and integrates multimodal data features.

[0032]

[0033] Where: Attention represents the text self-attention layer. LN represents the connection operation, LN represents the MLP layer, and σ represents the activation function. This indicates the extraction of overall text features. G represents the rich overall characteristics of the text's inherent information. Ti Represents a text scene diagram. and G represents the text and image scene diagrams after dimension adjustment, respectively. i This represents the constructed generalized scene graph.

[0034] The text self-attention layer has a sentence input size of 1×1×300, a triple input size of 1×3×300, and an output size of 1×1×300. The text scene graph has a dimension of 1×(T+1)×300, where T is the number of triples in a sentence, thus obtaining text features. The image scene graph has a dimension of 1×I×H×W, where I is the number of subject-object pairs in the image, and H and W are the height and width of the image feature map, respectively. The LN is an MLP consisting of two linear layers and two activation layers. The text linear layer has an input size of 300, a hidden layer dimension of 1024, and an output dimension of 2048. The image linear layer has an input dimension of H×W, a hidden layer dimension of 1024, and an output dimension of 2048.

[0035] Step 3, as follows Figure 3 As shown, the Transformer has 18 layers. The input to the Encoder layer is a text scene graph, which is then fused with intrinsic text information through an 8-head self-attention layer and a feedforward network to establish autocorrelation associations of text features. The input to the Decoder layer is the output of the Encoder layer and an image scene graph. The image scene graph is fused with intrinsic image information through an 8-head self-attention layer, and then combined with the output of the Encoder layer through an 8-head attention layer to associate the two modalities, performing cross-modal data fusion. Finally, the output is passed through a feedforward network, and privacy prediction is performed based on this result.

[0036] Step 4, as follows Figure 3 As shown, the feedforward network output obtained in step three is mapped to the privacy label dimension through a global average pooling layer and a linear layer, outputting the final result. Dimensions greater than 0.7 in the result are set to 1, and the loss function is calculated with the true label. The above steps are repeated until the loss function converges.

[0037] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal privacy leakage analysis framework based on generalized scene graphs, characterized in that, Includes the following steps: Step 1: Input the text and image into the corresponding scene graph feature extraction module for feature extraction, forming feature triples; Step 2: Enrich the overall feature information of the text by passing it through a self-attention layer, and then align it with the scene graph features of the image through an MLP (Multilayer Perceptron) layer to construct a generalized scene graph; Step 3: Input the generalized scene graph into the Transformer to model the relationship between features and privacy attributes and output the results. Step 4: Calculate the loss function for the output and the true labels, and repeat the above steps until the loss function converges.

2. The multimodal privacy leakage analysis framework based on generalized scene graphs according to claim 1, characterized in that, Step one includes the following steps: The input text data is plain English text, and the images are in JPG and JPEG formats. The image feature extraction module uses an attention-based scene graph generation method. First, a ResNet50 network extracts image features, which are then processed by a feature decoder and an entity decoder to obtain subject-object pairs. These pairs are then decoded into triples using a feedforward network layer and an attention layer. The text is then processed by a pre-trained text scene graph generation model, which extracts the text data into triples.

3. The multimodal privacy leakage analysis framework based on generalized scene graphs according to claim 1, characterized in that, Step two includes the following steps: It executes the rich overall features of the text's inherent information, performs unified representation of cross-modal data, and integrates features of multimodal data. Where: Attention represents the text self-attention layer. LN represents the connection operation, LN represents the MLP layer, and σ represents the activation function. This indicates the extraction of overall text features. G represents the rich overall characteristics of the text's inherent information. Ti Represents a text scene diagram. and G represents the text and image scene diagrams after dimension adjustment, respectively. i This represents the constructed generalized scene graph. The text self-attention layer has a sentence input size of 1×1×300, a triple input size of 1×3×300, and an output size of 1×1×300. The text scene graph has a dimension of 1×(T+1)×300, where T is the number of triples in a sentence, thus obtaining text features. The image scene graph has a dimension of 1×I×H×W, where I is the number of subject-object pairs in the image, and H and W are the height and width of the image feature map, respectively. The LN is an MLP consisting of two linear layers and two activation layers. The text linear layer has an input size of 300, a hidden layer dimension of 1024, and an output dimension of 2048. The image linear layer has an input dimension of H×W, a hidden layer dimension of 1024, and an output dimension of 2048.

4. The multimodal privacy leakage analysis framework based on generalized scene graphs according to claim 1, characterized in that, Step three includes the following steps: The Transformer has 18 layers. The input to the Encoder layer is a text scene graph, which is then fused with intrinsic text information through an 8-head self-attention layer and a feedforward network to establish autocorrelation associations of text features. The input to the Decoder layer is the output of the Encoder layer and an image scene graph. The image scene graph is fused with intrinsic image information through an 8-head self-attention layer, and then combined with the output of the Encoder layer through an 8-head attention layer to associate the two modalities, performing cross-modal data fusion. Finally, the output is passed through a feedforward network, and privacy prediction is performed based on this result.

5. The multimodal privacy leakage analysis framework based on generalized scene graphs according to claim 1, characterized in that, Step four includes the following steps: The feedforward network output obtained in step three is mapped to the privacy label dimension through a global average pooling layer and a linear layer, resulting in the final output. Dimensions greater than 0.7 in the result are set to 1, and the loss function is calculated using the ground truth labels. This process is repeated until the loss function converges.

Images