Private image detection method based on multi-scale features and cross-modal residual attention

Abstract

A privacy-preserving image detection method based on multi-scale features and cross-modal residual attention is proposed. In the offline stage, a privacy-preserving image-text dataset is constructed and a self-attention and cross-modal cross-attention residual network is trained. In the online stage, the trained network is used for real-time privacy-preserving image detection. This invention significantly improves the accuracy of privacy-preserving image detection through multi-scale features and residual cross-modal attention.

Description

Technical Field

[0001] This invention relates to a technology in the field of image recognition, specifically a privacy image detection method based on multi-scale features and cross-modal residual attention. Background Technology

[0002] Image privacy manifests in many ways. For example, photos containing identifiable facial or other physical features may leak personally identifiable information; photos containing geolocation data (such as GPS coordinates) may be used to track an individual's whereabouts or leak private information such as their home address; some photos may contain private scenes or situations, such as family gatherings, personal health information (e.g., medical photos), legal documents, etc., which should not be made public or accessed by others without authorization. The harm caused by image privacy breaches not only damages an individual's image and reputation but can also pose risks to personal safety. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a privacy image detection method based on multi-scale features and cross-modal residual attention. By utilizing multi-scale features and residual cross-modal attention, the accuracy of privacy image detection is significantly improved.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a privacy image detection method based on multi-scale features and cross-modal residual attention. In the offline stage, a privacy image text dataset is constructed and the constructed self-attention and cross-modal cross-attention residual network is trained. In the online stage, the trained network is used for real-time privacy image detection.

[0006] The aforementioned privacy-preserving image-text dataset includes vector representations of local images and word segments, as well as vector representations of global images and text, specifically obtained through the following methods:

[0007] Step 1: Obtain the complete labeled image from the public dataset.

[0008] Step 2: Preprocess the image from Step 1 by using the VinVL model to extract local images from the complete image and segment the text tags into words.

[0009] Step 3: Use the CLIP model to encode the local images and word segments to obtain the vector representation of each local image and word segment.

[0010] Step 4: Encode the global features using the ResNet model to obtain the vector representation of the global image; encode all text using the BERT model to obtain the vector representation of the global text.

[0011] The self-attention and cross-modal cross-attention residual network includes: an image self-attention network, a text self-attention network, a text-to-image attention residual network, an image-to-text attention residual network, two downsampling networks, and a feature fusion network. Specifically, the image self-attention network and the text self-attention network have the same structure and respectively use query feature vectors to calculate weights on the weight matrix and then weight them to obtain features. These features are then processed through normalization layers, fully connected layers, and residual connections to obtain the final single-modal image enhancement features and single-modal text enhancement features, respectively. The text-to-image attention residual network uses text query feature vectors to weight image key values. After weighting the image features by calculating and weighting the multiplication matrix, the text feature vector is obtained through a normalization layer, a fully connected layer, and a residual connection. The image-to-text attention residual network calculates and weights the text key-value weight matrix using the image query feature vector, and then obtains the text features through a normalization layer, a fully connected layer, and a residual connection to obtain the image feature vector. Two downsampling networks downsample the features output from the two corresponding attention residual networks through convolution operations to extract key features, reduce noise, and obtain two types of fused features. The feature fusion network obtains the classification result based on the two types of global features and the two types of fused features.

[0012] Technical effect

[0013] This invention employs a multi-scale feature extraction method, combined with various attention mechanisms to enhance features across multiple modalities. Through a downsampling network tailored to these attention mechanisms, it effectively captures and fuses local features while reducing unnecessary noise interference. A multi-modal feature fusion layer achieves efficient fusion of global and local features, further improving the model's overall performance and generalization ability. Compared to existing technologies, this invention significantly improves the accuracy of image privacy detection and reduces false positives. Attached Figure Description

[0014] Figure 1 This is a flowchart of the present invention;

[0015] Figure 2 This is a flowchart of a cross-modal cross-attention residual network. Detailed Implementation

[0016] like Figure 1 As shown, this embodiment relates to a privacy image detection method based on multi-scale features and cross-modal residual attention, including:

[0017] Step 1: Obtain the privacy image text dataset for training. The privacy image text dataset contains labeled data that includes privacy information and data that does not include privacy information. It includes images and corresponding text describing the images.

[0018] In the aforementioned privacy-preserving image-text dataset, the proportion of data containing privacy features is equal to the proportion of data not containing privacy features. The text length corresponding to each image may vary, but all images are guaranteed to have descriptive text.

[0019] Step 2: Preprocess the image and corresponding text and extract the global features of the image, global features of the text, local features of the image, and local features of the text with the same vector dimension.

[0020] The aforementioned global image features are obtained by inputting the image into a pre-trained ResNet network.

[0021] The aforementioned global text features are obtained by inputting the text content into the BERT network.

[0022] The image local features are obtained by using a VinVL network to select key target regions, which are strictly limited to 32. These key target regions are then input into a CLIP network for feature extraction, and the resulting feature vectors are concatenated.

[0023] The VinVL network described uses, but is not limited to, the techniques described by Pengchuan Zhang et al. in "VinVL: Revisiting Visual Representations in Vision-Language Models" to extract local images. The local images extracted by this technique need to be inspected; if the image size is smaller than 50*50 pixels, it is discarded.

[0024] The CLIP network described herein employs, but is not limited to, the techniques described by Alec Radford et al. in "Learning transferable visual models from natural language supervision" for preliminary feature extraction. The resulting features are then subjected to feature space dimensionality reduction.

[0025] The aforementioned feature space dimensionality reduction uses multiple dense weight matrices to perform a linear transformation on the input data, combined with a bias term, to achieve an effective mapping from the original high-dimensional space to the target low-dimensional space.

[0026] The aforementioned local text features are obtained by segmenting the text into words, sequentially inputting them into the CLIP network for feature extraction, and then concatenating the extracted features.

[0027] The aforementioned consistent vector dimension means that when the vector obtained from text segmentation is smaller than the dimension of the local features of the image, it is expanded to the same vector dimension by padding with zeros.

[0028] Step 3: After constructing and training the self-attention and cross-modal cross-attention residual network, the classification result is obtained based on the global image features, global text features, local image features, and local text features obtained in Step 2.

[0029] like Figure 2 As shown, the self-attention and cross-modal cross-attention residual network includes: an image self-attention network, a text self-attention network, a text-to-image attention residual network, an image-to-text attention residual network, two downsampling networks, and a feature fusion network.

[0030] The image-to-text attention residual network comprises: an image self-attention unit, a cross-attention unit, a residual unit, and a downsampling unit. Specifically: the image self-attention unit fuses information between local images using multiple matrices to obtain stronger information feature representations. The attention unit uses local image modal features as a basis, sequentially calculating the similarity of local labels as weights, and using the weight ratio of all labels to help strengthen local image features. The residual unit processes image features through a fully connected layer and then adds them to the original local features to prevent image feature degradation. The downsampling unit fuses global and local features of the image modality while reducing intra-modal noise.

[0031] The text-to-image attention residual network comprises: a text self-attention unit, an attention unit, a residual unit, a downsampling unit, and a feature fusion unit. Specifically: the text self-attention unit fuses information between local texts using multiple matrices to obtain stronger information feature representations. The attention unit uses local text modal features as a basis, sequentially calculating similarity as weights for local images, and using the weight ratios of all local images to help strengthen local text features. The residual unit processes the text features through a fully connected layer and then adds them to the original local features to prevent text feature degradation. The downsampling unit fuses global and local features of the text modality while reducing intra-modal noise.

[0032] The image feature vector is Where: the weight score of the i-th local text to the j-th local image For the i-th local text feature T i The j-th local image feature P j Text query feature vector These are the learnable weight matrices for the text features. These are the learnable weight matrices for the image features.

[0033] The text feature vector is Where: for the i-th local image feature P i The j-th local text feature T iImage query feature vector The weight score of the i-th local image to the j-th local text is:

[0034] The aforementioned unimodal text enhancement features and unimodal image enhancement features are both Wherein: the local features of each mode have three matrices W Q W V W K For the i-th local feature R i The j-th local feature R j Query feature Q i That is W Q ×R i The weight score of the i-th local feature to the j-th local feature is:

[0035] The downsampling mentioned above refers to: converting the feature output of the attention residual network into R. T and R Tl Stacked by columns Then, four similar downsampling layers are applied, each containing a convolution and a max-pooling operation. The pooling window size decreases from 36 to 33 sequentially, gradually reducing the width of the local text features from 36 to 32. During this process, four intermediate variables are generated. Concatenating these four variables yields a set of pooling results from different stages. Finally, the most suitable feature is adaptively selected through a fully connected layer to make a decision, resulting in two types of fused features.

[0036] The classification result is obtained by fusing global image features, global text features, and two types of fused features output by two downsampling networks through a feature fusion network. Specifically, the four features R are fused together. TL ,R PL ,R Tg ,R Pg After splicing, feature R is obtained. total =[R TL ,R PL ,R Tg ,R Pg After that, R is processed through the MLP layer. total The dimensions are reduced sequentially from 1624 to 256, 32, and 2. This allows for the obtaining of classification results after feature fusion.

[0037] The splicing process follows the order of local features first, then global features, making it easier for subsequent networks to extract information.

[0038] The MLP layer comprises: multiple fully connected layers, a ReLU activation function, and a stacked Dropout function, with the output being... After three rounds of iteration, Let k be the kth iteration. Where: k∈[1,3].

[0039] Through specific experiments, the above network was programmed using PyTorch, with the Adam optimizer selected, a learning rate of 0.0001 set, a training epoch of 150, and the cross-entropy function chosen as the loss function. The network was tested on the PrivacyAlert dataset (Zhao C, Mangat J, Koujalgi S, et al. Privacyalert: A dataset for image privacy prediction[C] / / Proceedings of the International AAAI Conference on Web and Social Media.2022,16:1352-1361), and the test results are shown in Table 1.

[0040] Table 1

[0041]

[0042]

[0043] Compared with existing technologies, this invention uses a multi-scale feature extraction method and designs a combination of multiple attention mechanisms to help enhance features across multiple modalities. By employing a downsampling network that aligns with the attention mechanism, it can effectively capture and fuse local features while reducing unnecessary noise interference. Furthermore, through a multi-modal feature fusion layer, it achieves efficient fusion of global and local features, further improving the overall performance and generalization ability of the model.

[0044] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A privacy-preserving image detection method based on multi-scale features and cross-modal residual attention, characterized in that, In the offline phase, a privacy image text dataset is constructed and a self-attention and cross-modal cross-attention residual network is trained. In the online phase, the trained network is used for real-time privacy image detection. The self-attention and cross-modal cross-attention residual network includes: an image self-attention network, a text self-attention network, a text-to-image attention residual network, an image-to-text attention residual network, two downsampling networks, and a feature fusion network. Specifically, the image self-attention network and the text self-attention network have the same structure and respectively use query feature vectors to calculate weights on the weight matrix and then weight them to obtain features. These features are then processed through normalization layers, fully connected layers, and residual connections to obtain the final single-modal image enhancement features and single-modal text enhancement features, respectively. The text-to-image attention residual network uses text query feature vectors to weight image key values. After weighting and summing the image features using the weighted matrix, a text feature vector is obtained through a normalization layer, a fully connected layer, and a residual connection. The image-to-text attention residual network uses the image query feature vector to weight the text key-value weight matrix and sums the text features, then obtains the image feature vector through a normalization layer, a fully connected layer, and a residual connection. Two downsampling networks, based on the feature outputs of their respective attention residual networks, perform convolution operations to downsampling, extracting key features, reducing noise, and obtaining two types of fused features. The feature fusion network then obtains the classification result based on the two types of global features and the two types of fused features. The image-to-text attention residual network includes: an image self-attention unit, a cross-attention unit, a residual unit, and a downsampling unit. Specifically: the image self-attention unit fuses information between local images using multiple matrices to obtain stronger information feature representation; the attention unit uses local image modal features as a basis, sequentially calculating the similarity of local labels as weights, and using the weight ratio of all labels to help strengthen local image features; the residual unit processes image features through a fully connected layer and then adds them to the original local features to prevent image feature degradation; the downsampling unit fuses global and local features of the image modality while reducing intra-modal noise. The text-to-image attention residual network includes: a text self-attention unit, an attention unit, a residual unit, a downsampling unit, and a feature fusion unit. Specifically: the text self-attention unit fuses information between local texts using multiple matrices to obtain stronger information feature representation; the attention unit uses local text modal features as a basis, sequentially calculating similarity as weights for local images, and using the weight ratios of all local images to help strengthen local text features; the residual unit processes text features through a fully connected layer and then adds them to the original local features to prevent text feature degradation; the downsampling unit fuses global and local features of the text modality while reducing intramodal noise.

2. The privacy image detection method based on multi-scale features and cross-modal residual attention as described in claim 1, characterized in that, The aforementioned privacy-preserving image-text dataset includes vector representations of local images and word segments, as well as vector representations of global images and text, specifically obtained through the following methods: Step 1: Obtain the complete labeled image from the public dataset; Step 2: Preprocess the image from Step 1 by using the VinVL model to extract local images from the complete image and segment the text tags into words. Step 3: Use the CLIP model to encode the local images and word segments to obtain the vector representation of each local image and word segment; Step 4: Encode the global features using the ResNet model to obtain the vector representation of the global image; encode all text using the BERT model to obtain the vector representation of the global text.

3. The privacy image detection method based on multi-scale features and cross-modal residual attention as described in claim 1, characterized in that, The image feature vector is Where: the weight score of the i-th local text to the j-th local image For the i-th local text feature The j-th local image feature Text query feature vector = ; , , These are the learnable weight matrices for the text features. , , These are the learnable weight matrices for image features; The text feature vector is Where: for the i-th local image feature The j-th local text feature Image query feature vector = The weight score of the i-th local image to the j-th local text is: ; The aforementioned unimodal text enhancement features and unimodal image enhancement features are both Where: the local features of each modality have three matrices , For the i-th local feature The j-th local feature Query features that is The weight score of the i-th local feature to the j-th local feature is: .

4. The privacy image detection method based on multi-scale features and cross-modal residual attention as described in claim 1, characterized in that, The downsampling mentioned above refers to: reducing the feature output of the attention residual network to... and Stacked by columns Then, through four similar downsampling layers, each containing a convolution and max pooling operation, the local text features with a width of 36 are gradually reduced to 32, generating intermediate variables. , , , The pooling results from different stages are then concatenated to obtain a set of pooling results. Then, the two types of fused features are obtained through adaptive selection by a fully connected layer.

5. The privacy image detection method based on multi-scale features and cross-modal residual attention according to claim 1, characterized in that, The classification result is obtained by fusing global image features, global text features, and two types of fused features output by two downsampling networks through a feature fusion network. Specifically, the four features are... Features are obtained after splicing Then, through the MLP layer... The dimensions are reduced from 1624 to 256, 32, and 2, respectively, so that the classification result is obtained after feature fusion.

6. The privacy image detection method based on multi-scale features and cross-modal residual attention according to claim 5, characterized in that, The MLP layer comprises: multiple fully connected layers, a ReLU activation function, and a stacked Dropout function, with the output being... After three rounds of iteration, Let k be the kth iteration. ,in: .

Images