Privacy image detection method based on multi-scale features and cross-modal residual attention

Abstract

The invention discloses a privacy image detection method based on multi-scale features and cross-modal residual attention, and the method comprises the steps: constructing a privacy image text data set at an offline stage, training a self-attention and cross-modal cross attention residual network obtained through construction, and carrying out the real-time privacy image detection through employing the trained network at an online stage. According to the method, the accuracy of privacy image detection is remarkably improved through multi-scale features and residual cross-modal attention.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image recognition, specifically a privacy image detection method based on multi-scale features and cross-modal residual attention. Background Art

[0002] Image privacy manifests itself in many ways. For example, when a photo contains identifiable personal facial or other body features, personal identity information may be leaked; when it contains geographic location data (such as GPS coordinates), it may be used to track a person's whereabouts or leak private information such as a person's home address; some photos may contain private scenes or situations, such as family gatherings, personal health information (such as medical photos), legal documents, etc. These contents should not be made public or accessed by others without authorization. The harm caused by image privacy leakage will not only damage personal image and reputation, but may even cause personal safety risks. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a privacy image detection method based on multi-scale features and cross-modal residual attention. The accuracy of privacy image detection is significantly improved through multi-scale features and residual cross-modal attention.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a privacy image detection method based on multi-scale features and cross-modal residual attention. In the offline stage, a privacy image text dataset is constructed and the constructed self-attention and cross-modal cross-attention residual network is trained. In the online stage, the trained network is used for real-time privacy image detection.

[0006] The privacy image and text dataset includes: vector representations of local images and word segments and vector representations of global images and texts, which are obtained in the following ways:

[0007] Step 1: Obtain complete images with labels from public datasets.

[0008] Step 2: Preprocess the image in step 1, use the VinVL model to extract the local image from the complete image, and cut the text label into segmented words.

[0009] Step 3: Use the CLIP model to encode the local images and word segmentations to obtain the vector representation of each local image and word segmentation.

[0010] Step 4: Use the ResNet model to encode the global features and obtain the vector representation of the global image; use the Bert model to encode all the text and obtain the vector representation of the global text.

[0011] The self-attention and cross-modal cross-attention residual network includes: an image self-attention network, a text self-attention network, a text-to-image attention residual network, an image-to-text attention residual network, two downsampling networks and a feature fusion network, wherein: the image self-attention network and the text self-attention network have the same structure and use the query feature vector to calculate the weight matrix and weight the features respectively, and obtain the final single-modal image enhancement features and single-modal text enhancement features respectively through the normalization layer, the fully connected layer and the residual connection; the text-to-image attention residual network uses the text query feature vector to calculate the image key weights. After the weights are calculated and weighted to obtain the image features, the text feature vector is obtained through the normalization layer, the fully connected layer and the residual connection; the image-to-text attention residual network uses the image query feature vector to calculate the weights of the text key-value weight matrix and weights to obtain the text features, and then the image feature vector is obtained through the normalization layer, the fully connected layer and the residual connection; the two downsampling networks are based on the feature outputs of the two corresponding attention residual networks, and the convolution operation is used to downsample, extract the key features and reduce the noise to obtain two types of fusion features; the feature fusion network obtains the classification results based on the two types of global features and the two types of fusion features. Technical Effects

[0012] The present invention uses a multi-scale feature extraction method, combined with multiple attention mechanisms to help multiple modalities perform feature enhancement, and effectively captures and fuses local features through a downsampling network that fits the attention mechanism, while reducing unnecessary noise interference. Through the multi-modal feature fusion layer, the efficient fusion of global features and local features is achieved, further improving the overall performance and generalization ability of the model. Compared with the prior art, the present invention significantly improves the accuracy of image privacy detection and reduces misjudgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the present invention;

[0014] Figure 2 Flowchart of the cross-modal attention residual network. DETAILED DESCRIPTION

[0015] like Figure 1 As shown, this embodiment relates to a privacy image detection method based on multi-scale features and cross-modal residual attention, including:

[0016] Step 1: Obtain a private image and text dataset for training. The private image and text dataset contains labeled data that contains privacy and does not contain privacy, including images and corresponding texts describing the images.

[0017] In the privacy image text dataset, the proportion of data containing privacy and data not containing privacy is the same. The length of the text corresponding to each image may be different, but it is ensured that there is a description text.

[0018] Step 2: Preprocess the image and the corresponding text and extract the image global features, text global features, image local features and text local features with consistent vector dimensions.

[0019] The global features of the image are obtained by inputting the image into a pre-trained ResNet network.

[0020] The global features of the text are obtained by inputting the text content into the Bert network.

[0021] The local features of the image are selected by the VinVL network to obtain the key target area, which is strictly stipulated to be 32. The key target area is input into the CLIP network to extract features in turn, and the output feature vectors are spliced.

[0022] The VinVL network uses, but is not limited to, the technology described in "VinVL: Revisiting Visual Representations in Vision-Language Models" by Pengchuan Zhang et al. to extract local images. The local images extracted by this technology need to be tested, and if the image size is less than 50*50 pixels, it is eliminated.

[0023] The CLIP network uses, but is not limited to, the technology described in "Learning transferable visual models from natural language supervision" by Alec Radford et al. to perform preliminary feature extraction. For the obtained features, the present invention performs feature space dimensionality reduction on them.

[0024] The feature space dimensionality reduction uses multiple dense weight matrices to perform linear transformation on the input data, combined with bias terms, to achieve effective mapping from the original high-dimensional space to the target low-dimensional space.

[0025] The local features of the text are obtained by cutting the text into words, inputting them into the CLIP network in sequence for feature extraction and splicing.

[0026] The vector dimension consistency means that when the vector obtained by text cutting is smaller than the dimension of the local feature of the image, it is expanded to the same vector dimension by zero filling.

[0027] Step 3: After constructing and training the self-attention and cross-modal cross-attention residual networks, the classification results are obtained based on the image global features, text global features, image local features, and text local features obtained in step 2.

[0028] like Figure 2 As shown, the self-attention and cross-modal cross-attention residual network includes: an image self-attention network, a text self-attention network, a text-to-image attention residual network, an image-to-text attention residual network, two downsampling networks and a feature fusion network.

[0029] The image-to-text attention residual network includes: an image self-attention unit, a cross-attention unit, a residual unit and a downsampling unit, wherein: the image self-attention unit obtains a stronger information feature expression by using multiple matrices to fuse the information between local images. The attention unit takes the local image modal features as the basis, calculates the similarity of the local labels in turn as weights, and helps to strengthen the local image features through the weight ratio of all labels. The residual unit processes the image features through the fully connected layer and then adds them to the original local features to prevent the image features from degrading. The downsampling unit fuses the global features and local features of the image modality, while reducing the noise within the modality.

[0030] The text-to-image attention residual network includes: a text self-attention unit, an attention unit, a residual unit, a downsampling unit and a feature fusion unit, wherein: the text self-attention unit obtains a stronger information feature expression by using multiple matrices to fuse the information between local texts. The attention unit takes the local text modal features as the basis, calculates the similarity of the local images in turn as weights, and helps to strengthen the local text features through the weight ratios of all local images. The residual unit processes the text features through the fully connected layer and then adds them to the original local features to prevent text feature degradation. The downsampling unit fuses the global features and local features of the text modality, while reducing the noise within the modality.

[0031] The image feature vector is Among them: the weight score of the i-th local text to the j-th local image For the i-th local text feature T i , the jth local image feature P j The text query feature vector are the learnable weight matrices of text features, are the learnable weight matrices of image features, respectively.

[0032] The text feature vector is Where: For the i-th local image feature P i , the jth local text feature T i, image query feature vector The weight score of the i-th local image to the j-th local text is:

[0033] The unimodal text enhancement features and unimodal image enhancement features are Among them: Each mode's local features have three matrices W Q ,W V ,W K , for the i-th local feature R i , the jth local feature R j , query feature Q i It's W Q ×R i The weight score of the i-th local feature to the j-th local feature is:

[0034] The downsampling mentioned above refers to: outputting the features of the attention residual network as R T and R Tl Stack by columns to get After that, it passes through 4 similar downsampling layers, each of which contains a convolution and maximum pooling operation. The pooling window size will be reduced from 36 to 33, and the local text feature with a width of 36 will be gradually reduced to 32. In this process, 4 intermediate variables will be generated Concatenate these four variables to get the pooling result sets at different stages Finally, the most appropriate features are adaptively selected through the fully connected layer to make decisions and obtain two types of fusion features.

[0035] The classification result is obtained by fusing the image global features, the text global features and the two types of fusion features output by the two downsampling networks through the feature fusion network. Specifically, the four features R TL ,R PL ,R Tg ,R Pg After splicing, we get feature R total =[R TL ,R PL ,R Tg ,R Pg ], R is transformed into total The dimension of is reduced from 1624 to 256, 32, and 2. Thus, the classification result is obtained after feature fusion.

[0036] The described splicing is in the order of first local features and then global features, so that the subsequent network can extract information more easily.

[0037] The MLP layer includes: multiple groups of fully connected layers, relu activation function, and Dropout function stacking, and its output is After 3 rounds of iterations, represents the kth iteration, then Where: k∈[1,3].

[0038] After specific experiments, the above network was programmed through Pytorch, the Adam optimizer was selected, the learning rate was set to 0.0001, the number of training epochs was set to 150, and the cross entropy function was selected as the loss function. The test was carried out on the PrivacyAlert dataset (Zhao C, Mangat J, Koujalgi S, et al. Privacy alert: A dataset for image privacy prediction [C] / / Proceedings of the International AAAI Conference on Web and Social Media. 2022, 16: 1352-1361). The test results are shown in Table 1.

[0039] Table 1

[0040] Compared with the prior art, the present invention adopts a multi-scale feature extraction method and designs a combination of multiple attention mechanisms to help multiple modalities perform feature enhancement. At the same time, through a downsampling network that fits the attention mechanism, it can effectively capture and fuse local features while reducing unnecessary noise interference and through the multi-modal feature fusion layer, it realizes efficient fusion of global features and local features, further improving the overall performance and generalization ability of the model.

[0041] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A private image detection method based on multi-scale features and cross-modal residual attention, characterized in that: In the offline stage, a private image text dataset is constructed and the constructed self-attention and cross-modal cross-attention residual network is trained. In the online stage, the trained network is used for real-time private image detection.

2. The method for private image detection based on multi-scale features and cross-modal residual attention according to claim 1, characterized in that: The privacy image and text dataset includes: vector representations of local images and word segments and vector representations of global images and texts, which are obtained in the following ways: Step 1: Obtain complete images with labels from public datasets; Step 2: Preprocess the image in step 1, extract the local image from the complete image using the VinVL model, and cut the text label into segmented words; Step 3: Use the CLIP model to encode the local image and word segmentation to obtain the vector representation of each local image and word segmentation; Step 4: Use the ResNet model to encode the global features and obtain the vector representation of the global image; use the Bert model to encode all the text and obtain the vector representation of the global text.

3. The method for private image detection based on multi-scale features and cross-modal residual attention according to claim 1, characterized in that: The self-attention and cross-modal cross-attention residual network includes: an image self-attention network, a text self-attention network, a text-to-image attention residual network, an image-to-text attention residual network, two downsampling networks and a feature fusion network, wherein: the image self-attention network and the text self-attention network have the same structure and use the query feature vector to calculate the weight matrix and weight the features respectively, and obtain the final single-modal image enhancement features and single-modal text enhancement features respectively through the normalization layer, the fully connected layer and the residual connection; the text-to-image attention residual network uses the text query feature vector to calculate the image key weights. After the weights are calculated and weighted to obtain the image features, the text feature vector is obtained through the normalization layer, the fully connected layer and the residual connection; the image-to-text attention residual network uses the image query feature vector to calculate the weights of the text key-value weight matrix and weights to obtain the text features, and then the image feature vector is obtained through the normalization layer, the fully connected layer and the residual connection; the two downsampling networks are based on the feature outputs of the two corresponding attention residual networks, and the convolution operation is used to downsample, extract the key features and reduce the noise to obtain two types of fusion features; the feature fusion network obtains the classification results based on the two types of global features and the two types of fusion features.

4. The method for private image detection based on multi-scale features and cross-modal residual attention according to claim 3, characterized in that: The image-to-text attention residual network includes: an image self-attention unit, a cross-attention unit, a residual unit and a downsampling unit, wherein: the image self-attention unit obtains a stronger information feature expression by using multiple matrices to fuse the information between local images; the attention unit takes the local image modal features as the basis, calculates the similarity of local labels in turn as weights, and helps to strengthen the local image features through the weight ratio of all labels; the residual unit processes the image features through the fully connected layer and then adds them to the original local features to prevent the image features from degrading; the downsampling unit fuses the global features and local features of the image modality and reduces the noise within the modality.

5. The method for private image detection based on multi-scale features and cross-modal residual attention according to claim 3, characterized in that: The text-to-image attention residual network includes: a text self-attention unit, an attention unit, a residual unit, a downsampling unit and a feature fusion unit, wherein: the text self-attention unit obtains a stronger information feature expression by using multiple matrices to fuse the information between local texts; the attention unit takes the local text modal features as the basis, calculates the similarity of the local images in turn as weights, and helps to strengthen the local text features through the weight ratios of all local images; the residual unit processes the text features through the fully connected layer and then adds them to the original local features to prevent the text features from degenerating; the downsampling unit fuses the global features and local features of the text modality and reduces the noise within the modality.

6. The method for private image detection based on multi-scale features and cross-modal residual attention according to claim 3, characterized in that: The image feature vector is Among them: the weight score of the i-th local text to the j-th local image For the i-th local text feature T i , the jth local image feature P j The text query feature vector are the learnable weight matrices of text features, are the learnable weight matrices of image features respectively; The text feature vector is Where: For the i-th local image feature P i , the jth local text feature T i , image query feature vector The weight score of the i-th local image to the j-th local text is: The unimodal text enhancement features and unimodal image enhancement features are Among them: Each mode's local features have three matrices W Q ,W V ,W K , for the i-th local feature R i , the jth local feature R j , query feature Q i It's W Q ×R i The weight score of the i-th local feature to the j-th local feature is:

7. The method for private image detection based on multi-scale features and cross-modal residual attention according to claim 3, characterized in that: The downsampling mentioned above refers to: outputting the features of the attention residual network as R T and R Tl Stack by columns to get After that, through 4 similar downsampling layers, each downsampling layer contains a convolution and maximum pooling operation, the local text features with a width of 36 are gradually reduced to 32 and the intermediate variables are generated. Then splice to get the pooling result set of different stages Then two types of fusion features are obtained through adaptive selection of the fully connected layer.

8. The method for private image detection based on multi-scale features and cross-modal residual attention according to claim 3, characterized in that: The classification result is obtained by fusing the image global features, the text global features and the two types of fusion features output by the two downsampling networks through the feature fusion network. Specifically, the four features R TL ,R PL ,R Tg ,R Pg After splicing, we get feature R total =[R TL ,R PL ,R Tg ,R Pg ], R is transformed into total The dimension is reduced from 1624 to 256, 32, and 2, respectively, so that the classification result is obtained after feature fusion.

9. The method for private image detection based on multi-scale features and cross-modal residual attention according to claim 8, characterized in that: The MLP layer includes: multiple groups of fully connected layers, relu activation function, and Dropout function stacking, and its output is After 3 rounds of iterations, represents the kth iteration, then Where: k∈[1,3].

Images