A Social Media Image Tampering Detection Method Based on Feature Enhancement Fusion

Abstract

This invention relates to a social media image tampering detection method based on feature enhancement fusion, used for forensic detection of tampered images disseminated through social media platforms. The invention mainly includes: (1) proposing a pyramid feature enhancement module that perceives multi-scale contextual information; and (2) proposing a social media image tampering detection framework based on feature enhancement fusion. Compared with existing technologies, this invention provides a social media image tampering detection method based on feature enhancement fusion, targeting the localization and detection of tampered images disseminated through social media platforms. The method of this invention is feasible and effective, and the trained model has high detection performance and real-time detection efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multimedia security technology, and particularly relates to a social media image tampering detection method based on feature enhancement fusion. BACKGROUND

[0002] With the rapid popularization of mobile Internet and the rapid development of information technology, digital images have become one of the important media for people to convey information at present. At the same time, with the rapid spread of multimedia information technology, various image editing software has also emerged as the times require, for example, the wide application of image editing software such as Meitu, Photoshop, etc. makes image processing more simple. Both professional users and ordinary users can use these image editing tools to edit images, and the edited images are almost invisible to the naked eye. Although most people only use these image editing software for entertainment activities, some unscrupulous people also edit images illegally to achieve their own purpose. In the era of rapid development of information technology, various social media platforms also grow explosively, such as Facebook, WeChat, Weibo, Whatsapp, etc. The rise of these social software simplifies the process of sharing and spreading pictures. More and more people learn, work and live through these social media software. Social media software has become the main platform for information dissemination, and digital images occupy an important part. The image after the forgery will seriously affect people's views after being spread through social media platforms. Therefore, how to detect the tampered images spread through social media is a problem to be solved.

[0003] In order to better adapt to the actual demand, in recent years, there are more and more image tampering detection methods for composite operation type. The literature "Wu Y, AbdAlmageed W, Natarajan P. ManTra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA. 2019: 9543-9552." proposes a self-supervised fully convolutional network ManTra-Net, which is an end-to-end network that can classify 385 types of image operations. At the same time, this article considers the image tampering detection problem as a local anomaly detection problem, designs a z-score feature to capture anomalies, and proposes a new LSTM-based method to evaluate abnormal areas. The literature "Wang J, Wu Z, Chen J, et al. Objectformer for image manipulation detection and localization / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. New Orleans, Louisiana, USA. 2022: 2364-2373." models the visual consistency of the object level through the Transformer, combines the high-frequency features of the image, and introduces the block embedding of the image frequency domain information and the RGB domain. The use of a learnable object-level representation as an intermediate representation refines the block embedding to capture consistency.The document "Wu H, Zhou J, Tian J, Zhou J, Liu J. Robust Image Forgery Detection Against Transmission Over Online Social Networks. IEEE Transactions on Information Forensics and Security, 2022, 17: 443-456." first realizes image tampering detection based on social media platforms. This article divides the noise generated by social media platforms into foreseeable noise and unforeseeable noise, simulates the noise in the social media scene using a deep neural network, and thus can locate and detect tampered images that have been transmitted through social media platforms.

[0004] Most tampered images in real life are transmitted through social media platforms, but current tampered image forensics algorithms do not consider tampered images transmitted through social media. Compared with tampered images that have not been transmitted through social media, tampered images transmitted through social media will cover up the original tampering traces and increase the difficulty of forensic detection. Considering the problem of practical application, the present application aims to improve the detection performance and real-time detection efficiency of tampered images transmitted through social media, and proposes a social media image tampering detection method based on feature enhancement fusion. SUMMARY

[0005] The present application proposes a social media image tampering detection method based on feature enhancement fusion, mainly including two major contents:

[0006] (1) A pyramid feature enhancement module for multi-scale perception of context information is proposed;

[0007] (2) A social media image tampering detection framework based on feature enhancement fusion is proposed.

[0008] The specific contents are as follows:

[0009] (1) A pyramid feature enhancement module for multi-scale perception of context information is proposed: the structure of the module is as Figure 1The module obtains the multi-scale receptive field information of the image through four branches. In order to capture the context information of different scales, the four branches respectively adopt different hole rates and convolution kernels, wherein the hole rates are respectively set as [1, 6, 12, 18], and the convolution kernel sizes are 1x1, 3x3, 3x3 and 3x3. However, due to the large hole rate, the long-range information of the image is obtained, and the long-range information of the image is usually some insignificant information. In view of this problem, a channel attention mechanism is added behind the four branches, which captures the explicit relationship between channels in this way, so that the model responds to the largest spatial features while ignoring the small spatial features, so as to make full use of the image information contained in the four branches. In addition, the branches with small hole rates and the branches with large hole rates are combined through add operation to realize the fusion of image information, so that the network learns the multi-scale context information. Finally, the fused tensor is spliced and passed through a 3x3 convolution layer and a 1x1 convolution layer to add the original input features. The purpose of this is to fuse the global image information and the context information of different scales, and finally suppress the irrelevant regions through the spatial attention mechanism.

[0010] (2) A social media image tampering detection framework based on feature enhancement fusion is proposed. The overall framework is as shown in Figure 2 The shallow features of the image have high resolution, rich spatial information and contain more edge contour information; the deep features have rich semantic information and pay more attention to the global information of the image, and each pixel point represents the information of a region, but some details are lost. Based on this, the image features extracted by the backbone network are divided into shallow features and deep features. Considering the detection performance and detection efficiency requirements of the tampered image in the social media scene, the backbone network used when extracting the abstract features of the image is Resnet101, and the first layer is used as the shallow feature and the last three layers are used as the deep feature.

[0011] Because ordinary convolution may cause problems such as loss of internal data structure and loss of spatial hierarchical information, it is not conducive to extracting rich image features in image detection work. Therefore, in the backbone network, instead of using ordinary convolution, a dilated convolution is used, and its calculation formula is

[0012]

[0013] Wherein, x(m, n) represents input, y(m, n) represents output, w(i, j) represents a convolution kernel with length and width of M and N respectively, r represents a hole rate, the number of zero elements filled between adjacent convolution kernel elements is r-1, and when r=1, there is no zero element filled between adjacent convolution kernel elements, at this time, the hole convolution is ordinary convolution. The hole convolution can expand the receptive field of the image under the premise that the feature map size is unchanged, thereby improving the detection effect of the tampered image. Therefore, in the backbone network, the hole convolution with a hole rate of 1, 1, 1 and 2 is adopted.

[0014] In order to improve the detection performance of the detection network on tampered images of different sizes, the image information extracted by the shallow features and the deep features is respectively passed through the pyramid feature enhancement module of the multi-scale perception context information, and then the three deep features of the backbone network are subjected to add operation. The operation can not only obtain more rich image features, but also can reduce the calculation amount and improve the detection efficiency.

[0015] After the shallow features and the deep features of the image are extracted, the model fuses the features extracted by the backbone network. First, a 1x1 convolution layer is used to adjust the channel number of the shallow feature branch, and then the shallow feature branch is fused with the deep feature after two times of upsampling through the channel dimension tensor splicing method. The upsampling method used here is bilinear interpolation. The fused features are passed through two 3x3 convolution layers and a 1x1 convolution layer, and finally the network output is adjusted to the input image size through four times of upsampling. Similarly, the upsampling method used here is also bilinear interpolation, and the final output of the network is the prediction result.

[0016] The loss function uses the predicted value and the true value of the network to measure the performance of the network and reflects the fitting effect of the network on the target task. The loss function used by the model can be represented as

[0017]

[0018] Wherein, L represents the value of the loss, N is the number of data sets during training, C is the number of classifications, y ij is the value of the true label, p ij is the value of the predicted pixel.

[0019] Compared with the prior art, the above technical scheme has at least the following beneficial effects:

[0020] 1. The multi-scale perception context information pyramid feature enhancement module provided by the application. Four branches with different hole rates and convolution kernels are used to capture multi-scale context information. Channel attention mechanisms are added behind the four branches to capture the display relationship between channels. In addition, the branches with smaller hole rates and the branches with larger hole rates are combined through add operation to realize image information fusion, so that the network learns multi-scale context information. Finally, the fused tensor is spliced, passed through a 3x3 convolution layer, and added with the original input features through a 1x1 convolution layer. Finally, the obtained features pass through the spatial attention mechanism to suppress irrelevant regions. The multi-scale perception context information pyramid feature enhancement module mentioned in the scheme can obtain the features of tampered images of different sizes, and improve the model detection performance.

[0021] 2. The social media image tampering detection framework based on feature enhancement fusion provided by the application. The shallow features of an image contain more pixel information, such as image color, texture, edge, and corner information, while the deep features contain some more abstract information. The image features are divided into shallow features and deep features. First, the shallow features and the deep features of the image are obtained respectively, and then they are respectively passed through the designed pyramid feature enhancement module with hole convolution to extract the perception context information of the image. Finally, the shallow information and the deep information are fused and put into the decoder, and are up-sampled to the original image size, so as to obtain the detection result of the tampered image. Compared with existing detection models, the application has more robust detection performance and higher real-time detection efficiency for tampered images spread through social media, and can have higher application value in actual deployment. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The multi-scale perception context information pyramid feature enhancement module of the application is a schematic diagram.

[0023] Figure 2 The social media image tampering detection framework based on feature enhancement fusion of the application is a schematic diagram. DETAILED DESCRIPTION

[0024] The application is a social media image tampering detection distortion method based on feature enhancement fusion.

[0025] The development language is python, the development environment is Linux, the deep learning framework is pytorch, the training set uses Dresden as the original image, splices it with the objects in the dataset MS-COCO to generate tampered images, and randomly selects 9984 images from them to participate in network training. The training set verification set ratio is 9:1. The test set is Columbia, DSO, NIST, and the image sets obtained by transmitting them through Facebook, WeChat, Weibo, and Whatsapp respectively.

[0026] Step 1: Prepare the dataset.

[0027] The dataset Dresden is used as the original image, then it is spliced with the objects in the dataset MS-COCO to generate tampered fake images, from which 9984 are selected as the training set for network training. The datasets Columbia, DSO, and NIST are used as test sets, and they are transmitted through Facebook, WeChat, Weibo, and Whatsapp respectively, and each dataset gets a tampered dataset that has been transmitted through the four social media platforms. The final test set is composed of the original test set and the datasets that have been transmitted through the four social media platforms.

[0028] Step 2: Build the network structure.

[0029] The network structure is shown in Figure 2 , the tampered image passes through the backbone network Resnet101 to obtain the shallow and deep features of the image, and then the shallow and deep features pass through the feature enhancement module and are fused, and then are upsampled to the image size by the decoder.

[0030] Step 3: Network training.

[0031] The model is built according to the design shown in Figure 2 for end-to-end training. The initial learning rate is set to 1e-3, the loss function is shown in formula (2), and the goal of the model is to minimize the loss value. The SGD optimizer is used for training, the batch size of the input image is 12, a total of 50 rounds of training are performed, and the total number of iterations is 37400 times. During training, the performance indicator is calculated every 10 epochs, and if the indicator is greater than the current maximum value, the current model is saved as the best performance model. In addition, the last training model is also saved at the end of network training.

[0032] Step 4: Model testing.

[0033] The step 4 inputs are test sets of Columbia, DSO, NIST and image sets obtained after transmission through Facebook, WeChat, Weibo and WhatsApp. The output is the tampered area of the tampered image. F1-score, IoU and AUC, three commonly used pixel-level evaluation indicators, are used for performance evaluation, and the IoU threshold of positive and negative samples is set to 0.5. The time unit s is used as an evaluation indicator of the algorithm efficiency. Through the above evaluation indicators, it can be judged that the present application has high detection performance and detection efficiency.

[0034] In summary, the present application designs a social media image tampering detection method based on feature enhancement fusion, which has good social media image detection performance and high real-time detection efficiency, meeting the requirements of fake image detection in actual scenarios.

[0035] Those skilled in the art can understand that the protection scope of the present application is not limited to the specific embodiments described. Those skilled in the art can make equivalent changes or replacements to related technical features without departing from the principles of the present application. It should be noted that the technical solutions after the changes or replacements will fall within the protection scope of the present application.

Claims

1. A method for social media image forgery detection based on feature enhancement fusion, characterized in that, The method comprises: The encoder, the pyramid feature enhancement module of multi-scale perception context information, comprises four hollow convolution branches with different hollow rates and convolution kernels: the hollow rates of the four branches are [1, 6, 12, 18] respectively, the convolution kernel sizes are 1*1, 3*3, 3*3 and 3*3 respectively, then a channel attention mechanism is added behind each branch to capture the display relationship between channels; since the shallow feature of the image has a small receptive field and a large resolution, it can process and save small-scale geometric features; while the high-level feature has a large receptive field and a small resolution, it can process and save large-scale semantic features, therefore, in order to capture more context information of different scales, the branch with a smaller hollow rate is combined with the branch with a larger hollow rate through an add operation, the information fusion of image geometric features and semantic features is realized, the image positioning accuracy in a complex scene is improved, finally, the fused tensor is spliced, then a 3*3 convolution layer is used and the original input feature is added through a 1*1 convolution layer, finally, the obtained feature is suppressed in an irrelevant area through spatial attention; The decoder divides the tampered image features extracted by the backbone network into shallow features and deep features, learns image edge information by using the characteristics of shallow features and deep features learning high-level semantic information, extracts rich image information, then the image information extracted by the shallow features and the deep features is respectively subjected to the pyramid feature enhancement module of multi-scale perception context information for capturing multi-scale context perception information, finally, the image information extracted by the shallow features and the deep features is fused and up-sampled to the size of the image by the decoder; The encoder adopts resnet101 as a backbone network, takes the first layer of resnet101 as a shallow feature, takes the following three layers as deep features, and performs an add operation on the three deep features, so that the image feature information is increased, thereby obtaining more abundant image features. In addition, in the backbone network, the article respectively adopts a dilated convolution with a dilated rate of 1, 1, 1 and 2, and the dilated convolution calculation formula is, wherein, represents input, represents output, represents length, width is , a convolution kernel, represents a dilated rate, the number of zero elements filled between adjacent convolution kernel elements is , and when , there is no zero element filled between adjacent convolution kernel elements, at this time, the dilated convolution is a normal convolution.

2. The feature enhancement fusion based social media image forgery detection method of claim 1, wherein, The decoder specifically comprises: The shallow feature branch divided in the backbone network is subjected to a 1*1 convolution layer, then the shallow feature branch is fused with the deep feature subjected to two times of up-sampling through the channel dimension tensor splicing mode, the up-sampling mode adopted here is bilinear interpolation, the fused feature is subjected to two 3*3 convolution layers and a 1*1 convolution layer, finally, the network output is adjusted to the input image size through four times of up-sampling, similarly, the up-sampling mode adopted is also bilinear interpolation, and finally, the prediction result of the network is output.

Images