Social media privacy analysis framework based on multi-modal relation extraction

Abstract

The invention relates to a social media privacy analysis framework based on multi-modal relation extraction, and belongs to the technical field of artificial intelligence and privacy analysis. The existing privacy protection technology mostly focuses on single-mode data processing and is difficult to deal with the privacy leakage problem in multi-mode data interaction. According to the method, through a multi-modal relation extraction technology, cross-modal semantic alignment, knowledge graph construction and bidirectional mutual information optimization are fused, privacy analysis is modeled as a relation extraction task, a'subject-relation-object 'triple is extracted to model a privacy relation, and a privacy leakage mechanism in multi-source and multi-modal data is effectively identified. The framework can be integrated into a social platform content auditing system or an intelligent privacy risk assessment tool, and the potential privacy leakage risk is accurately detected and analyzed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and privacy analysis technology, and specifically to a privacy analysis method based on multimodal relation extraction (MRE). This method utilizes multimodal relation extraction technology to innovatively integrate cross-modal semantic alignment, knowledge graph construction, and bidirectional mutual information optimization to accurately detect and analyze potential privacy leakage risks caused by the interaction of multi-source and multimodal data (text, images, etc.) in social media. This technology can be integrated into application scenarios such as social platform content review systems and intelligent privacy risk assessment tools. Background Art

[0002] With the rapid development of social media, user-generated multimodal content, including text, images, and videos, has exploded. This surge in data presents new opportunities for information mining and understanding, while also posing significant challenges to traditional privacy protection technologies. Existing privacy protection mechanisms primarily focus on anonymizing or obfuscating single-modal data, such as anonymizing text or obfuscating images. However, these methods are unable to effectively address privacy leaks arising from the interaction of multi-source and multimodal data. Cross-modal analysis of multimodal data can reveal privacy attributes that cannot be detected through single-modal analysis. For example, analyzing a conference room photo or a text mentioning a "client meeting" separately may not reveal sensitive information, but analyzing them together could reveal private information such as organizational affiliations.

[0003] Existing privacy analysis technologies have the following major limitations. First, single-modal analysis has obvious shortcomings. Textual privacy analysis mainly relies on semantic and temporal pattern analysis, or uses large language models (LLMs) to infer personal attributes from text, but it has difficulty resolving ambiguity in the absence of visual context. Visual privacy analysis, on the other hand, infers social relationships through scene understanding or graph-based reasoning, but often ignores textual evidence. Image analysis alone may not accurately identify the identity or relationship type of people in the image. Whether it is text or visual privacy analysis, existing methods are mostly limited to single-modal analysis and cannot detect new privacy relationships generated through multimodal interactions.

[0004] To address this challenge, this paper models privacy analysis as a multimodal relation extraction (MRE) task. The leakage of private relations is usually manifested as implicit relation triples (e.g.<LILI,couple with,COLE> ). Therefore, the goal of privacy analysis is to map these cross-modal dependencies into<subject,relation,object> Relation triples. However, existing multimodal relation extraction (MRE) methods also face many challenges when applied to privacy analysis. First, existing multimodal relation extraction (MRE) methods are usually text-dominated and ignore the importance of visual information. This text-dominated modal bias leads to poor performance in privacy inference where visual information dominates. Second, multimodal data in the real world often has weak alignment problems, such as text and images are partially related or complementary. Such weakly aligned multimodal data may lead to semantic ambiguity in privacy analysis, thereby missing the extraction of private information. In addition, effective privacy analysis requires the combination of external knowledge, such as domain-specific constraints and common sense reasoning, but existing multimodal privacy analysis methods lack the ability to reason about implicit privacy relations and cannot effectively utilize external knowledge to enhance the accuracy of privacy analysis.

[0005] To address the above problems, this paper proposes a multimodal privacy analysis framework (PAMRE), which models privacy analysis as a multimodal relation extraction (MRE) task. It models privacy relations by extracting "subject-relation-object" triples, thereby systematically identifying privacy leakage mechanisms in multi-source multimodal data. Summary of the Invention

[0006] This paper proposes a multimodal privacy analysis framework (PAMRE), which aims to systematically identify and analyze private information in multimodal social media data using multimodal relation extraction (MRE). This framework models privacy analysis as a relation extraction task, modeling private relationships by extracting "subject-relation-object" triplets. This effectively addresses the issue of privacy leakage in multimodal data interactions.

[0007] In order to overcome the limitations of the existing technology, the present invention introduces a dual-channel alignment mechanism, which solves the problem of weak inter-modal alignment through a visual semantic alignment module and a text semantic alignment module. In the visual semantic alignment module, a pre-trained multimodal large language model (such as QianWen) is used to generate descriptive subtitles to capture the fine-grained semantic information of the image. At the same time, ResNet50 is used to extract block-level and region-level features of the image, and BERT is used to embed the subtitles at the word and phrase levels. Through the cross-attention mechanism, the subtitles and image features are aligned to ensure the semantic consistency of the visual and text modalities. In the text semantic alignment module, the Stable Diffusion model is used to generate images based on text prompts to enrich the semantic representation of the text. Multi-granularity semantic alignment is performed on the generated images and the original text to further enhance the alignment effect between modalities. This dual-channel alignment mechanism not only solves the problem of weak inter-modal alignment, but also ensures the semantic consistency of visual and text modalities in privacy analysis.

[0008] In order to enhance the reasoning ability of implicit relationships, the present invention constructs a cross-modal knowledge graph. The construction of a cross-modal knowledge graph mainly includes the following steps: First, extract seed concepts from text and images, such as extracting subjects and objects from text through named entity recognition, and extracting significant concepts from images through pre-trained concept detection models. Then, retrieve knowledge paths from external knowledge bases such as ConceptNet, connect the seed concepts in text and images through knowledge paths, and construct a cross-modal knowledge graph. Finally, use a multi-head graph attention network (GAT) to process the knowledge graph and extract global semantic information. The construction of this cross-modal knowledge graph not only enhances the model's reasoning ability for implicit relationships, but also improves the depth and breadth of privacy analysis.

[0009] In order to highlight the equal importance between modalities, the present invention introduces a two-way mutual information maximization strategy. By maximizing the semantic correspondence and reducing cross-modal noise, the alignment effect between modalities is optimized. The noise contrast estimation (NCE) framework is adopted, and other representations are used as negative samples to calculate the mutual information (MI) loss. At the same time, the reverse mutual information maximization step is introduced to ensure that the visual and textual modalities contribute equally in the shared representation, thereby improving the robustness of the model. This two-way mutual information optimization strategy not only optimizes the alignment effect between modalities, but also reduces noise interference, thereby improving the robustness and accuracy of the model.

[0010] In summary, this paper models privacy analysis as a multimodal relationship extraction task for the first time. Through the above-mentioned technological innovations, it effectively addresses the limitations of existing privacy analysis technologies and provides a new solution for privacy analysis of multi-source and multimodal data in social media. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is the overall architecture diagram of the present invention, showing the collaborative work of the visual semantic alignment module (ViSAlign), the text semantic alignment module (TeSAlign), the cross-modal knowledge graph feature extractor (CrossGraph), and the representation fusion and prediction module.

[0012] Figure 2 Schematic diagram of the limitations of traditional models in reasoning about privacy relations.

[0013] Figure 3 Schematic diagram of multimodal fusion in privacy relationship reasoning of the present invention. DETAILED DESCRIPTION

[0014] The multimodal privacy analysis framework (PAMRE) of the present invention is implemented through the following specific implementation steps: Figure 1 As shown in Figure 2, the framework covers data preprocessing, detailed operations of each module, and model training and optimization processes.

[0015] 1. Visual Semantic Alignment Module (ViSAlign)

[0016] The visual semantic alignment module enhances the semantic consistency of visual and textual modalities by generating descriptive captions and aligning them with image features. The specific steps are as follows:

[0017] First, we use the pre-trained multimodal large language model Qianwen to generate descriptive captions for the image. Given an input image I, the model generates captions C by prompting prompt = {Please give me a detailed description of the picture}. This process can be formalized as:

[0018] C=MLLM(prompt,I)

[0019] MLLM stands for Multimodal Large Language Model, and prompt stands for the generation instruction.

[0020] Next, in order to align the subtitles with the image features, ResNet50 is used to extract the block-level features of the image (I s ) and regional characteristics (I r ). At the same time, BERT is used to perform word-level (C w ) and phrase level (C p) embedding. The cross-attention mechanism is used to align captions with image features, with the text embedding T as the query and the visual embedding V as the key and value. The cross-attention calculation formula is:

[0021]

[0022] Among them, d is the dimension of text embedding, W d is the learnable attention matrix.

[0023] During the alignment process, the fine-grained representations (G f ) and coarse-grained representation (G c Fine-grained representation of aligned captions with word-level embeddings (C w ) and the block-level features of the image (I s ), while the coarse-grained representation aligns the phrase-level embeddings of the captions (C p ) and the regional features of the image (I r ). Combine these two representations into a comprehensive representation G, namely:

[0024] G=G f +G c

[0025] This multi-granularity semantic alignment ensures that the semantic consistency of visual and textual modalities is modeled at different levels, thus providing richer semantic information for private relation extraction.

[0026] 2. Text Semantic Alignment Module (TeSAlign)

[0027] The text semantic alignment module generates images that are semantically consistent with the input text and aligns these image features with the text features to further enhance the semantic consistency between modalities. The specific implementation steps are as follows:

[0028] First, the Stable Diffusion model is used to generate an image V based on the input text T. The model generates an image that is highly consistent with the text semantics through a diffusion process, which can be formalized as follows:

[0029] V=LDMs(T)

[0030] Among them, LDMs represents diffusion models.

[0031] Next, the generated image V and the original text T are semantically aligned in a multi-granular manner. In terms of visual encoding, the same ResNet50 encoder as the ViSAlign module is used to extract the block-level features of the image (V s ) and regional characteristics (V rIn terms of text encoding, BERT is used to perform word-level (T w ) and phrase level (T p ) embedded.

[0032] The alignment of text and image features is also achieved through the cross-attention mechanism, where the text embedding T is used as the query and the visual embedding V is used as the key and value. The calculation formula of the cross-attention module is similar to that of the ViSAlign module:

[0033]

[0034] During the alignment process, the fine-grained representations (H f ) and coarse-grained representation (H c ). Fine-grained representation of word-level embedding T of aligned text w With the block-level features V of the image s , while the coarse-grained one represents the phrase-level embedding T of the aligned text p and the region-level features V of the image r The two representations are combined into a comprehensive representation H, namely:

[0035] H=H f +H c

[0036] This multi-granularity semantic alignment ensures that the semantic consistency of text and image modalities is modeled at different levels, thereby providing more accurate semantic alignment information for private relation extraction.

[0037] 3. Cross-modal Knowledge Graph Feature Extractor (CrossGraph)

[0038] The cross-modal knowledge graph feature extractor integrates external knowledge to enhance the model's ability to reason about implicit relationships. The specific implementation steps are as follows:

[0039] First, named entity recognition (NER) is used to extract subject and object pairs from the text and annotate their semantic types (such as person, location, and organization). A pre-trained concept detection model is used to extract salient concepts (such as "person" and "university") from the image.

[0040] We then retrieve knowledge paths in ConceptNet, using the extracted seed concepts as the starting and ending points. These paths represent the semantic relationships between concepts. For each pair of text and image seed concepts, we retrieve the shortest path (no more than 10 nodes in length). To capture bidirectional semantic dependencies, we construct paths not only from text concepts to image concepts, but also from image concepts to text concepts in the reverse direction. This bidirectional path helps the model gain a more comprehensive understanding of the relationships between concepts.

[0041] Finally, the retrieved paths are integrated into a unified knowledge graph, where nodes represent concepts and edges represent semantic relationships. A multi-head graph attention network (GAT) is used to process the knowledge graph and aggregate neighborhood information by calculating attention weights between nodes. For each node i, the attention weight between it and its neighboring node j is calculated as:

[0042] e ij =SelfAttention(W g g i , W g g j )

[0043] Among them, W g is the learnable weight matrix, g i and g j are the embedding vectors of nodes i and j respectively. The attention weights are normalized by the softmax function:

[0044] α ij =softmax(e ij )

[0045] Update the representation of each node according to the attention weight:

[0046]

[0047] Where σ is the activation function, W k is a learnable weight matrix. This process is repeated in multiple attention heads to capture semantic relationships in different aspects. Finally, the updated node features are aggregated into global semantic information K for subsequent privacy relationship prediction:

[0048]

[0049] Where N is the total number of nodes in the knowledge graph.

[0050] Through the above steps, the cross-modal knowledge graph feature extractor (CrossGraph) not only integrates the semantic information of text and images, but also enhances the model's ability to reason about implicit relationships through the external knowledge base ConceptNet. This knowledge graph construction and feature extraction process provides strong semantic support for the accurate prediction of private relationships.

[0051] 4. Representation Fusion and Prediction Module

[0052] The representation fusion and prediction module fuses multimodal features with knowledge graph features and predicts private relationships. The specific implementation steps are as follows:

[0053] First, the representation G extracted by the ViSAlign module, the representation H extracted by the TeSAlign module, the representation K extracted by the CrossGraph module, and the features of text entities and noun phrases are fused to obtain the final fused representation F. The specific formula is:

[0054] F=T s +T o +P+G+H+K

[0055] Among them, T S and T o denote the embeddings of subject and object respectively, and P denotes the aggregated features of noun phrases.

[0056] Next, a nonlinear transformation is performed on the fusion representation F to enhance the model’s expressive power. The specific formula is:

[0057] F fused =Dropout(ReLU(W2·ReLU(W1·F+b1)+b2))

[0058] Among them, W1, W2, b1, and b2 are learnable parameters.

[0059] Finally, the transformed representation F fused Input to the prediction layer, and use the softmax function to calculate the probability distribution of the privacy relationship. The specific formula is:

[0060] P(y|F uused )=softmax(W p ·F fused +b p )

[0061] Among them, W p and b p is the learnable parameter of the prediction layer, and y represents the predicted relationship category.

[0062] 5. Maximizing Bidirectional Mutual Information

[0063] To further optimize the alignment between modalities, this paper introduces a two-way mutual information maximization strategy, whose working logic is similar to the "back-translation verification" in translation quality assessment:

[0064] (1) Forward verification channel: Generate descriptive text based on image content to verify whether the visual information accurately conveys the text semantics.

[0065] (2) Reverse verification channel: Generate a corresponding image based on the text description and evaluate the semantic consistency between the generated image and the original image.

[0066] This mechanism is implemented through noise contrast learning technology. During the model training process:

[0067] (1) Construct a positive sample set: a pair of truly matched image and text data.

[0068] (2) Construct a negative sample set: a randomly shuffled image and text combination.

[0069] By maximizing the mutual information of positive samples and minimizing the correlation of negative samples, the model can gain accurate cross-modal understanding capabilities.

[0070] The embodiment demonstrates the ability of the present invention to accurately infer private relationships by fusing visual and textual modalities. Figure 2 As shown in Figure 2, traditional text-only models (such as BERT) may only be able to extract shallow relationships from text data, such as (LILI, attend_event_with, COLE). Due to the lack of visual context, these models cannot capture the intimate nature of the relationship. On the other hand, although pure visual methods can detect spatial associations, such as (Person_A, stand_close_with, Person_B), they cannot identify the specific relationship type. To solve this problem, the present invention adopts the PAMRE framework, such as Figure 3 As shown, the present invention generates descriptive titles based on visual input. For example, it can generate a title such as "a sophisticated couple, dressed in elegant formal wear, standing close together", linking the visual proximity of the two people with the "together" mentioned in the text. In addition, the present invention also integrates ConceptNet to enrich the relationship reasoning process. ConceptNet enhances the connection between visual context and text data by introducing scene-related privacy concepts (such as "wedding" and "ceremony"). Through this multimodal fusion, the present invention can accurately predict the "couple" relationship. This embodiment demonstrates that the present invention significantly improves the accuracy of privacy relationship reasoning compared to traditional pure text or pure visual methods.

[0071] Through the above detailed technical implementation steps, the multimodal privacy analysis framework (PAMRE) of the present invention can effectively identify and analyze privacy leakage mechanisms in multimodal data on social media. After actual measurement, the privacy recognition rate of this solution on the MNRE dataset reached 87.39%, an improvement of 26.53% over the unimodal baseline method. This method can be integrated into the privacy review system of social platforms or used in compliance detection tools to automatically identify potential privacy risks in user-uploaded content. Those skilled in the art can make appropriate adjustments and modifications to the specific implementation details without departing from the core concept of the present invention.

Claims

1. A social media privacy analysis framework based on multimodal relationship extraction, characterized by: include: The visual semantic alignment module ViSAlign is used to enhance the semantic consistency of visual and textual modalities by generating descriptive captions and aligning them with image features; the text semantic alignment module TeSAlign is used to further enhance the semantic consistency between modalities by generating images that are semantically consistent with the input text and aligning these image features with text features; the cross-modal knowledge graph feature extractor CrossGraph is used to enhance the model's reasoning ability for implicit relationships by integrating external knowledge; the representation fusion and prediction module is used to fuse multimodal features and knowledge graph features and perform privacy relationship prediction.

2. The social media privacy analysis framework based on multimodal relationship extraction according to claim 1 is characterized in that: The visual semantic alignment module ViSAlign includes: using the pre-trained multimodal large language model QianWen to generate descriptive captions for images, using ResNet50 to extract block-level and region-level features of the image, using BERT to embed the generated captions at the word and phrase levels, and achieving multi-granularity alignment of captions and image features through a cross-attention mechanism.

3. The social media privacy analysis framework based on multimodal relationship extraction according to claim 1 is characterized in that: The text semantic alignment module TeSAlign includes: using a Stable Diffusion model to generate an image based on the input text, using ResNet50 to extract block-level and region-level features of the generated image, using BERT to embed the input text at the word and phrase levels, and achieving multi-granularity alignment of text and image features through a cross-attention mechanism.

4. The social media privacy analysis framework based on multimodal relationship extraction according to claim 1 is characterized in that: The cross-modal knowledge graph feature extractor CrossGraph includes: extracting subject and object pairs from text through named entity recognition technology, extracting salient concepts from images through a pre-trained concept detection model, retrieving knowledge paths in ConceptNet, constructing a cross-modal knowledge graph, using a multi-head graph attention network (GAT) to process the knowledge graph, and extracting global semantic information.

5. The social media privacy analysis framework based on multimodal relationship extraction according to claim 1 is characterized in that: The representation fusion and prediction module includes: fusing the features extracted by the visual semantic alignment module, the text semantic alignment module and the cross-modal knowledge graph feature extractor, performing nonlinear transformation on the fused features, and using the softmax function to calculate the probability distribution of the privacy relationship.

6. The social media privacy analysis framework based on multimodal relationship extraction according to claim 1 is characterized in that: It also includes a two-way mutual information maximization strategy to optimize the alignment effect between modalities and reduce cross-modal noise.

Images