Method and device for carrying out tampering detection on digital image and storage medium

Abstract

The invention relates to the technical field of information security, and discloses a method and a device for tampering detection of a digital image, and a storage medium, the method comprises the following steps: obtaining a camera fingerprint of a to-be-detected image, the to-be-detected image being an image carrying tampering traces, the camera fingerprint being determined based on a pre-trained camera fingerprint extraction model, the camera fingerprint is used for representing camera characteristics of a camera of the to-be-detected image, inputting the to-be-detected image and the camera fingerprint into a pre-trained tampering detection model for detection, and obtaining a pixel region corresponding to the tampering trace in the to-be-detected image and a tampering probability of the to-be-detected image, according to the scheme, the pixel areas where various types of tampering traces are located in the to-be-detected image can be accurately detected by using the tampering detection model, and the tampering probability of the to-be-detected image is obtained, so that the tampering detection precision is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of information security technology and provides a method, device and storage medium for detecting tampering of digital images. Background Art

[0002] As a medium of communication, digital images contain a lot of very important information. However, with the increasing popularity of image editing software, people can easily and diversely tamper with images. Tampered images may even be used illegally, thus posing a serious threat to social security.

[0003] For some tampered images, especially when the tampered images come from different acquisition devices, or the tampered images are transmitted through social media, the various lossy operations and external processing processes that the tampered images may undergo in their life cycle are not considered, resulting in the inability to have good robustness against digital images with different lossy processing operations. Summary of the invention

[0004] Embodiments of the present application provide a method, device, and storage medium for detecting tampering of a digital image, so as to improve the accuracy of tampering detection of tampering traces in an image to be detected.

[0005] The specific technical solutions provided by this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for detecting tampering of a digital image, comprising:

[0007] Obtaining a camera fingerprint of an image to be detected, wherein the image to be detected is an image carrying traces of tampering, and the camera fingerprint is determined based on a pre-trained camera fingerprint extraction model, and the camera fingerprint is used to characterize the camera characteristics of the camera of the image to be detected;

[0008] The image to be detected and the camera fingerprint are input into the pre-trained tampering detection model for detection, and the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected are obtained.

[0009] Optionally, the image to be detected is obtained by:

[0010] Acquire the image to be detected based on the original image; or

[0011] Acquire the image to be detected based on the original image and the social transmission medium; or

[0012] Acquire the image to be detected based on the original image and the pre-trained transmission noise model;

[0013] Among them, the original image is taken by a camera with camera fingerprint.

[0014] Optionally, obtaining the image to be detected based on the original image and the social transmission medium includes:

[0015] Uploading the original image to a social transmission medium for display, so that the social transmission medium adds transmission noise to the original image;

[0016] The original image after adding transmission noise is downloaded from the social transmission media, and the original image after adding transmission noise is used as the image to be detected.

[0017] Optionally, obtaining an image to be detected based on an original image and a pre-trained transmission noise model includes:

[0018] Inputting the original image into a pre-trained transmission noise model for learning, so that the transmission noise model outputs the original image with simulated transmission noise, wherein the simulated transmission noise is obtained from the output of the original transmission noise model trained as the transmission noise model;

[0019] The original image with simulated transmission noise is used as the image to be detected.

[0020] Optionally, obtaining a camera fingerprint of the image to be detected includes:

[0021] The image to be detected is input into the pre-trained camera fingerprint extraction model for detection to obtain the camera fingerprint corresponding to the image to be detected.

[0022] Optionally, the image to be detected and the camera fingerprint are input into a pre-trained tampering detection model for detection, and the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected are obtained, including:

[0023] Input the image to be detected and the camera fingerprint into the pre-trained tampering detection model, and use the encoder of the tampering detection model to encode the image to be detected and the camera fingerprint respectively to obtain a first encoding feature corresponding to the image to be detected and a second encoding feature corresponding to the camera fingerprint;

[0024] Using the cross fusion module of the tampering detection model to fuse the first coding feature and the second coding feature to obtain a fusion feature;

[0025] The decoder of the tampering detection model is used to decode the fused features to obtain the pixel area corresponding to the pixel unit corresponding to the tampering trace in the image to be detected;

[0026] The classification head of the tampering detection model is used to extract fusion features, and the tampering probability of the image to be detected is obtained based on the fusion features.

[0027] Optionally, before the encoder of the tampering detection model encodes the image to be detected and the camera fingerprint respectively, it also includes:

[0028] Use the tampering detection model to pre-convolute the camera fingerprint so that the dimension of the obtained camera fingerprint is the same as the dimension of the image to be detected;

[0029] The cross fusion module of the tampering detection model is used to fuse the first coding feature and the second coding feature to obtain a fusion feature, including:

[0030] The cross-fusion module in the tampering detection model is used to perform cross-convolution fusion on the first encoding feature and the second encoding feature to obtain a fused feature.

[0031] In a second aspect, an embodiment of the present application further provides a device for detecting tampering of a digital image, comprising:

[0032] An acquisition unit, used for acquiring a camera fingerprint of an image to be detected, wherein the image to be detected is an image carrying traces of tampering, the camera fingerprint is determined based on a pre-trained camera fingerprint extraction model, and the camera fingerprint is used to characterize the camera characteristics of the camera of the image to be detected;

[0033] The detection unit is used to input the image to be detected and the camera fingerprint into a pre-trained tampering detection model for detection, and obtain the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected.

[0034] Optionally, the image to be detected is obtained by:

[0035] Acquire the image to be detected based on the original image; or

[0036] Acquire the image to be detected based on the original image and the social transmission medium; or

[0037] Acquire the image to be detected based on the original image and the pre-trained transmission noise model;

[0038] Among them, the original image is taken by a camera with camera fingerprint.

[0039] Optionally, the image to be detected is acquired based on the original image and the social transmission medium, and the acquisition unit is used to:

[0040] Uploading the original image to a social transmission medium for display, so that the social transmission medium adds transmission noise to the original image;

[0041] The original image after adding transmission noise is downloaded from the social transmission media, and the original image after adding transmission noise is used as the image to be detected.

[0042] Optionally, the image to be detected is acquired based on the original image and a pre-trained transmission noise model, and the acquisition unit is used to:

[0043] Inputting the original image into a pre-trained transmission noise model for learning, so that the transmission noise model outputs the original image with simulated transmission noise, wherein the simulated transmission noise is obtained from the output of the original transmission noise model trained as the transmission noise model;

[0044] The original image with simulated transmission noise is used as the image to be detected.

[0045] Optionally, the camera fingerprint of the image to be detected is obtained, and the obtaining unit is used to:

[0046] The image to be detected is input into the pre-trained camera fingerprint extraction model for detection to obtain the camera fingerprint corresponding to the image to be detected.

[0047] Optionally, the image to be detected and the camera fingerprint are input into a pre-trained tampering detection model for detection, and the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected are obtained. The detection unit is used to:

[0048] Input the image to be detected and the camera fingerprint into the pre-trained tampering detection model, and use the encoder of the tampering detection model to encode the image to be detected and the camera fingerprint respectively to obtain a first encoding feature corresponding to the image to be detected and a second encoding feature corresponding to the camera fingerprint;

[0049] Using the cross fusion module of the tampering detection model to fuse the first coding feature and the second coding feature to obtain a fusion feature;

[0050] The decoder of the tampering detection model is used to decode the fused features to obtain the pixel area corresponding to the pixel unit corresponding to the tampering trace in the image to be detected;

[0051] The classification head of the tampering detection model is used to extract fusion features, and the tampering probability of the image to be detected is obtained based on the fusion features.

[0052] Optionally, before the encoder of the tampering detection model encodes the image to be detected and the camera fingerprint respectively, it also includes:

[0053] Use the tampering detection model to pre-convolute the camera fingerprint so that the dimension of the obtained camera fingerprint is the same as the dimension of the image to be detected;

[0054] The cross fusion module of the tampering detection model is used to fuse the first coding feature and the second coding feature to obtain a fusion feature, including:

[0055] The cross-fusion module in the tampering detection model is used to perform cross-convolution fusion on the first encoding feature and the second encoding feature to obtain a fused feature.

[0056] According to a third aspect, a computing device includes:

[0057] A memory for storing executable instructions;

[0058] A processor is used to read and execute executable instructions stored in a memory to implement any method as described in the first aspect.

[0059] In a fourth aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any method described in the first aspect.

[0060] The beneficial effects of this application are as follows:

[0061] In summary, in an embodiment of the present application, a method, device and storage medium for tampering detection of a digital image are provided, the method comprising: obtaining a camera fingerprint of an image to be detected, wherein the image to be detected is an image carrying traces of tampering, the camera fingerprint is determined based on a pre-trained camera fingerprint extraction model, the camera fingerprint is used to characterize the camera characteristics of the camera of the image to be detected, the image to be detected and the camera fingerprint are input into a pre-trained tampering detection model for detection, and the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected are obtained. In this solution, the tampering detection model is used to accurately detect the pixel areas where various types of tampering traces are located in the image to be detected and obtain the tampering probability of the image to be detected, thereby effectively improving the accuracy of tampering detection.

[0062] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0064] Figure 1 A schematic diagram of a training process of a transmission noise model in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of a detection process of a transmission noise model in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of a training process of a camera fingerprint extraction model in an embodiment of the present application;

[0067] Figure 4A schematic diagram of a detection process of a camera fingerprint extraction model in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of a training process of a tamper detection model in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of a detection process of a tamper detection model in an embodiment of the present application;

[0070] Figure 7 A schematic diagram of the composition of a cross-fusion module in a tamper detection model in an embodiment of the present application;

[0071] Figure 8 A schematic diagram of a process for detecting tampering of a digital image in an embodiment of the present application;

[0072] Fig. 9 A schematic diagram of a process for obtaining an image to be detected based on an original image and a social transmission medium in an embodiment of the present application;

[0073] Fig.10 A schematic diagram of a process for obtaining an image to be detected based on an original image and a pre-trained transmission noise model in an embodiment of the present application;

[0074] Fig.11 A schematic diagram of the process of obtaining the camera fingerprint of the image to be detected in an embodiment of the present application;

[0075] Fig.12 A schematic diagram of a flow chart of encoding an image to be detected and a camera fingerprint respectively by using an encoder of a tamper detection model in an embodiment of the present application;

[0076] Fig.13 This is a flow chart of using the pre-convolution and cross-fusion modules to encode the image to be detected and the camera fingerprint respectively in an embodiment of the present application;

[0077] Fig.14 This is a flow chart of using the cross fusion module of the tamper detection model in the embodiment of the present application to fuse the first coding feature and the second coding feature to obtain a fused feature;

[0078] Fig.15 A schematic diagram of the logical architecture of a device for detecting tampering of a digital image in an embodiment of the present application;

[0079] Fig.16 A schematic diagram of the physical architecture of a computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0081] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented using sequences other than those illustrated or described herein.

[0082] First, some terms in the embodiments of the present application are explained as follows:

[0083] Image tampering refers to adding or removing important features from an image without leaving any obvious traces.

[0084] Camera fingerprint: A technique that identifies and distinguishes different camera devices by analyzing and extracting the unique features produced by the camera when taking a photo. These features are usually determined by the camera's hardware and software settings, including sensor characteristics, lens characteristics, image processing algorithms, etc. In the process of digital image forensics, the camera fingerprint of a photo is analyzed to verify the source of the image.

[0085] Currently, digital images are marked by a series of subtle and invisible traces during their lifecycle, which can come from many different sources, from the camera's internal processing steps when the camera captures the image, to all the external processing that the image encounters during its lifecycle (e.g., recompression by social media, resizing, adding filters, etc.). When an image is tampered with, these obvious traces can be destroyed, resulting in inconsistent traces.

[0086] Especially when the images come from different acquisition devices or when the images are transmitted through social media, the existing detection technologies have poor detection effects. The main reason is that these technologies do not consider the various lossy operations that images may undergo during their life cycle and the characteristics of different acquisition devices, resulting in the inability to accurately detect digital images with different lossy processing operations.

[0087] The preferred implementation modes of the present application are described in detail below with reference to the accompanying drawings.

[0088] In the embodiment of the present application, the system includes at least one computing device, and the implementation of the method for detecting tampering of a digital image is mainly performed on the computing device side.

[0089] First, the training process of the transmission noise model, camera fingerprint extraction model and tampering detection model used in the embodiments of the present application is introduced in detail.

[0090] (I) Transmission noise model

[0091] The transmission noise model is trained by the original transmission noise model, which is usually trained by a deep neural network with an encoding-decoding architecture (such as a U-net model, a Swin-Transformer model, etc.). During the training process, the input of the original transmission noise model is the original image set, and the output of the original transmission noise model is the predicted noise image set. The original image set includes multiple original images, the predicted noise image set includes predicted noise images corresponding to multiple original images, and the real noise image set includes lossy images obtained by displaying social transmission media corresponding to multiple original images. During the training process, the loss function calculated between the predicted noise image corresponding to each original image and the obtained lossy image is used for model convergence. In this way, the trained transmission noise model can better simulate the transmission noise of social transmission media. After the original image is input into the model, the output predicted noise image will have a consistent distribution with the lossy image actually transmitted through the social transmission media, which is conducive to the downstream training of a robust tampering detection model.

[0092] It should be noted that the original images in the above original image set are from public datasets such as COCO and ImageNet, and the lossy images in the above real noise image set are obtained by uploading and downloading images from public datasets such as COCO and ImageNet through social transmission media. Considering the large number of datasets, Python scripts were used to write scripts during the training process, and the requests library was used to make network requests to realize the automatic upload and download process. In addition, before model training, the size of each image was unified to 256×256.

[0093] Since the output of the transmission noise model is a lossy image with the same resolution as the input image, the trained transmission noise model needs to satisfy the image-to-image mapping relationship, that is, for the input original image x∈R H ×W×C , the goal is to learn a function mapping Ω: R^(H×W×C)->R^(H×W×C) so that the output of the model is the predicted noise image y∈R H×W×CMost networks based on the encoder-decoder architecture can meet the above requirements. The present invention uses U-net based on Resnet34 as the transmission noise model, that is, the encoder is Resnet34 with the fully connected layer removed, and the decoder is consistent with U-net (using upsampling and convolutional layer blocks to fuse encoding features and decoding features), such as Figure 1 and Figure 2 It should be noted that the model structure is replaceable, and most networks that satisfy the image-to-image mapping relationship can meet the requirements.

[0094] During the training process, the loss function Loss1 is calculated using formula (1):

[0095] Loss1=∑ x∈X ||Ω(x)-y||2 Formula (1)

[0096] Among them, X represents the original image set, x represents an original image, Ω represents the mapping function of the transmission noise model, y represents the real noise image corresponding to x, and ||·||2 represents the L2 norm. During the training process, when the value of the loss function Loss1 no longer decreases, the model converges, and the trained transmission noise model can better fit the real distribution of network transmission images in real situations.

[0097] (II) Camera fingerprint extraction model

[0098] Cameras or other imaging devices (hereinafter collectively referred to as cameras) often perform multiple processing operations when outputting images, such as data compression, interpolation, and gamma correction, etc., but different cameras implement them in different ways. These processing operations will cause the camera to leave unique traces (hereinafter collectively referred to as camera fingerprints) when outputting each image. When the image is tampered with, these unique fingerprints may be destroyed, resulting in inconsistencies in the camera fingerprint. If this situation can be detected, effective forensic analysis can be performed.

[0099] In order to effectively extract the camera fingerprint of the image and capture inconsistent fingerprint traces, that is, those that may be tampered with, the present invention trains a camera fingerprint extraction model to model the camera fingerprint unique to each image. During the training process, the input of the original camera fingerprint extraction model is a set of image patch pairs, and the output of the original camera fingerprint extraction model is a set of camera fingerprint pairs. The image patches are all cropped from the original image. The camera fingerprint pair set includes the camera fingerprints corresponding to two paired image patches. During the training process, each paired original image patch is passed through the camera fingerprint model, and the loss function calculated by the output camera fingerprint pair is used for model convergence.

[0100] The original camera fingerprint extraction model uses a lightweight deep neural network with 15 trainable layers, which receives a three-channel input image and outputs a single-channel camera fingerprint image. Image patch pairs are randomly cropped from the dataset from different camera models for model training. The trained camera fingerprint extraction model will have the following characteristics: for patch pairs from the same camera model and the same pixel position, the model will output the same camera fingerprint; conversely, for any image matching pair with different attributes, different camera fingerprints will be output. Therefore, each input raw image is deterministically associated with a camera fingerprint image that enhances the spatial distribution of camera model artifacts and their model dependencies.

[0101] The above image patch pair set can come from three public datasets: DSO-1, MFC18 and NIMBLE16, and the dataset needs to be cropped at a specified position to obtain a patch image set, and then any two images in the patch image set are matched to obtain a patch pair. It should be noted that the images in these datasets come from different cameras and have different image formats. Therefore, it can be regarded as a very challenging benchmark dataset for the model to be trained. In order to allow the camera fingerprint model to be trained based on contrast loss, the labels corresponding to the above input (image patch pair) are defined as follows during the training process: 1) Positive label matching pair: when the two patch images are from the same camera and their relative positions in the image are the same; 2) Negative label matching pair: the two patch images are not from the same camera, or the two patch images are from the same camera but their relative positions in the image are different.

[0102] The training process of the camera fingerprint extraction model is as follows Figure 3 The purpose of the camera fingerprint extraction model is to extract its unique noise fingerprint from each image. It emphasizes all traces related to the camera's internal processing steps. As mentioned above, it is an important forensic clue indicating tampering. Consistent with previous research on camera fingerprint extraction models, the camera fingerprint extraction model used in the training process of this invention is also a deep neural network model, and its network structure is as follows: Figure 4 As shown. The model satisfies the image-to-image mapping, that is, for the input image x∈R H×W×C , the goal is to learn a function mapping R H×W×C ->R H×W×C , so that the output of the model is y∈R H×W×1 , that is, the output of the model is a camera fingerprint with the same resolution as the input image and the number of channels is 1. Figure 3As shown, in order to train the model, the image from the training set is first cropped at a specified position to obtain a set of image patches. Any two image patches are combined from the above image patch set, and the true labels are defined for them (the definition method is as described in the previous paragraph), thereby forming a set of image patch pairs. The two images of an image patch pair are input into the original camera fingerprint extraction model. They share the weight parameters of the model. The two outputs are used to calculate the loss function, and back propagation is performed to update the weight parameters of the model. The purpose of updating the model parameters is to make the trained model meet the following characteristics: for image patch pairs with positive labels, the model outputs two identical camera fingerprints; and for image patch pairs with negative labels, two different camera fingerprints are output.

[0103] In order to make the camera fingerprint extraction model meet the above conditions, the present invention designs the loss function Loss2 of the model based on the InfoNCE contrast loss. The contrast value is calculated by formula 2, and the loss function Loss2 is calculated by formula 3.

[0104] d i,j =||Φ(i)-Φ(j)|| 2 Formula (2)

[0105]

[0106] Where Φ(·) represents the mapping function of the original camera fingerprint extraction model; ||xy|| 2 represents the square of the Euclidean distance between x and y; X represents the set of image patches; G i represents the subset of patches with the same camera origin and the same pixel positions as the i-th patch.

[0107] The calculation of the above loss function enables the model to learn the similarity of input patch pairs using the idea of ​​contrastive learning. Therefore, the effectiveness of the model in extracting camera fingerprints can be improved. During the training phase, for positively labeled patch pairs, the model updates the weights to reduce the distance between the outputs; while for negatively labeled patch pairs, the input patch pairs are not similar, so the model updates the weights to increase the distance between the outputs. When the value of the loss function Loss2 no longer decreases, the model converges and a trained camera fingerprint extraction model is obtained. Finally, for each input image, the trained camera fingerprint extraction model outputs a specific camera fingerprint that emphasizes all traces related to the camera's internal processing steps.

[0108] (III) Tamper Detection Model

[0109] Usually, the images to be detected are all lossy processed by social transmission media. The noise introduced by these lossy operations (hereinafter referred to as transmission noise) may affect the effectiveness of the tampering detection method. In fact, most of the images to be detected are from network transmission media. In order to enhance the immunity of the tampering detection model to transmission noise and improve the robustness of detection, the present invention trains a dual-branch fusion tampering detection model for detecting and locating tampering traces in the image to be detected. The model has two input ports, which are used to input the lossy image and the camera fingerprint corresponding to the image.

[0110] During the training process, the input data of the original tampering detection model is the damaged image and its corresponding camera fingerprint. The output of the original tampering detection model is the predicted tampering area and the predicted tampering probability. The above-mentioned damaged image is an image carrying transmission noise, and the camera fingerprint is the camera fingerprint corresponding to the damaged image. During the training process, the loss function calculated based on the predicted tampering area and the actual tampering area, as well as the predicted tampering probability and the actual tampering label is used for model convergence.

[0111] Exemplarily, the above-mentioned lossy image with transmission noise is obtained by inputting the original image into a trained transmission noise model, and the camera fingerprint is obtained by inputting the lossy image into a camera fingerprint extraction model. By adding transmission noise in the training stage, the tampering detection model can learn more general features transmitted through social transmission media, thereby significantly improving the detection performance of the tampering detection model. By injecting the camera fingerprint as an auxiliary input to the tampering detection model, the sensitivity to weak and critical tampering features can be enhanced. As an important forensic information, the camera fingerprint records the tampering traces of the camera's internal processing steps. Considering that once the camera fingerprint has been tampered with, it will lead to inconsistent tampering traces, it can provide the model with subtle but critical tampering clues. In addition, multimodal input helps the network learn rich tampering features, thereby enhancing the model's ability to locate the tampered area.

[0112] The above-mentioned lossy image set can use images from datasets such as CASIA and Columbia. These datasets involve different types of tampering, such as image splicing, copy-move and removal. It should be noted that the CASIA dataset consists of two versions, namely CASIA1.0 and CASIA2.0, and CASIA2.0 is used for the above-mentioned training process. In addition, in order to make up for the shortcomings of the training dataset, the Dresden dataset can be used as the source of the original image, and the forged image can be generated by splicing the original image with the objects in the MS-COCO dataset, and the forged image is used as a supplement to the lossy image set. In addition to the image itself, all datasets also contain the corresponding pixel-level tampered area, which is a binary image with a tampered area value of 1 and a background area value of 0. In order to make the model output not only the pixel-level tampered area, but also the image-level tampering probability, it is necessary to supplement each image with an image-level true label, namely the true tampering probability, which is defined as follows: 0 represents that the image has not been tampered with, and 1 represents that the image has been tampered with.

[0113] The training process of the tamper detection model is shown in Figure 5 First, the original image x is input into the trained transmission noise model, and a lossy image μ is output that is close to the real social transmission medium. Then, the lossy image μ is input into the trained camera fingerprint extraction model, and a specific camera fingerprint υ is output. Finally, the lossy image μ and the camera fingerprint υ are input into the tampering detection model for training, aiming to detect tampered images with image-level accuracy and locate tampered areas with pixel-level accuracy. That is, for the input image {μ∈R H×W×C ,υ∈R H×W×1}, the model aims to learn a function mapping Θ, so that the output of the model contains an image-level tampering probability value p∈[0,1] and a pixel-level tampering area ο∈R H×W×1 , ο is a tampering probability image, and each pixel value represents the probability value of the pixel predicted by the model as tampered. That is, the model satisfies the mapping Θ: {R H×W×C , R H×W×1}->{R H×W×1 , p}.

[0114] The schematic diagram of the tamper detection model is shown in Figure 6. As can be seen from the figure, the input of the network is the lossy image μ and its corresponding camera fingerprint υ. First, in order to align the shapes of the two inputs, a 3×3 convolution operation is performed on the camera fingerprint υ so that the number of its output channels is 3. Secondly, the lossy image μ and the camera fingerprint υ after the convolution operation are respectively fed to the encoder for feature extraction. It should be noted that the encoder used in the present invention is Resnet34 after the fully connected layer is removed. After the encoding is completed, two encoding features {F_rgb, F_cam} will be obtained. These two encoding features come from input images of different modalities and therefore have features of different scales. In order to fuse the above-mentioned features from different modal paths, a cross-fusion module is designed during the training process. The cross-fusion module fuses the information of the dual-mode features through a symmetrical dual path. Finally, the fused features are respectively input into the decoder path and the classification head for pixel-level tampering area positioning and image-level tampering probability prediction. It should be noted that in order to achieve pixel-level positioning, the fused features are input into the decoder path for decoding. The decoder of this model is the same as the decoder used in the transmission noise model, which is based on operations such as convolution, upsampling and splicing to fuse the encoding-decoding features. Starting from the fused features output by the cross-fusion module, it is fused with the encoding features from the same resolution through convolution and upsampling. Because the fused features effectively fuse the features of the camera fingerprint and the lossy image, the encoding features fused in the decoder only consider the encoding features from the lossy image. In addition, in order to achieve image-level prediction, the fused features are input into the classification head for classification prediction, which contains two fully connected layers. The number of output nodes of the first fully connected layer is 512, and the number of output nodes of the second fully connected layer is 2.

[0115] The above cross-fusion module is implemented based on a symmetrical dual path, and its structure is as follows: Figure 7 As shown. Taking one of the paths as an example, first, the input feature F∈R H,W,C A 1×1 convolution is used in parallel to perform channel compression to reduce the amount of calculation and obtain two compressed features. In order to perform subsequent cross attention, the above compressed features are transformed into a shape of R N,C1 At the same time, the other path also obtained two embedding features of shape R N,C1The four embedded features and the original two inputs are input into the cross-attention module together, so as to fully exchange information between multiple modalities. The cross-attention module uses the features extracted from one of the modalities to correct the information from the other modality, which helps to use the knowledge of the other modality to filter out the noise information of a certain modality, and obtain the corrected features of the two modalities {F1, F2}. The above output features are concatenated and fused, and then aggregated using convolution operations, and finally merged into a single output feature map F_merge. The output feature map F_merge and all the encoded features of the lossy image μ represent the input of the decoder, which is used to generate the final tampering area positioning map.

[0116] During the training process, the objective function is designed based on the binary cross entropy loss. First, for the two outputs of the model {ο∈R H×W×1 , p} and the given true label {δ∈R H×W×1 , y}, two loss functions Loss3 and Loss4 are calculated respectively, where Loss3 is the objective function of the tampering localization task, and Loss4 is the objective function of the tampering detection task. The loss function Loss3 is calculated by formula 4, and the loss function Loss4 is calculated by formula 5. Secondly, in order to achieve joint optimization, the final objective function Loss5 of the network is calculated by formula 6.

[0117]

[0118] Loss4 = - ((1-y)log(1-p)+ylogp) Formula 5

[0119] Loss5 = λLoss3 + (1- λ)Loss4 Formula 6

[0120] Wherein, N represents the total number of pixels in an image; δ_i and ο_i represent the actual tampering probability of the i-th pixel and the predicted tampering probability that the model regards it as a tampered area, respectively; α and β are weight parameters introduced to solve the problem of class imbalance, and the present invention sets them to 0.5 and 2 respectively according to empirical values; y represents the real image-level label of the image, and p represents the predicted value that the network regards the image as a tampered image; λ is the weight introduced to calculate Loss3 and Loss4, and the present invention sets it to 0.5 according to empirical values.

[0121] After introducing the model training process, the following will continue to introduce the process of using the trained model to detect tampering of digital images. Figure 8 As shown, in an embodiment of the present application, a specific process of detecting tampering of a digital image is as follows:

[0122] Step 201: Obtain a camera fingerprint of an image to be detected, wherein the image to be detected is an image carrying traces of tampering, and the camera fingerprint is determined based on a pre-trained camera fingerprint extraction model, and the camera fingerprint is used to characterize the camera characteristics of the camera of the image to be detected.

[0123] The image to be detected may be a transmission image from different social media or from different cameras. The image to be detected may carry transmission noise or may not carry transmission noise. The image to be detected may carry traces of tampering or may not carry traces of tampering. In the embodiment of the present application, the image to be detected with traces of tampering is mainly detected.

[0124] During the detection process, when it is necessary to perform tampering detection on the image to be detected, that is, it is necessary to determine the pixel area corresponding to the tampering traces in the image to be detected, and it is necessary to determine the tampering probability of the image to be detected being tampered with, considering the influence of transmission noise, it is necessary to first determine the original image corresponding to the image to be detected. In the embodiment of the present application, the image taken by the camera or other acquisition device (hereinafter collectively referred to as the camera) is the original image. During the implementation process, after the original image is determined, first use method one, method two or method three to obtain the image to be detected. Method one: obtain the image to be detected based on the original image. Or

[0125] In this way, the original image taken by the camera is directly used as the image to be detected. Obviously, the image to be detected here does not carry any transmission noise or tampering traces.

[0126] Method 1: Obtain the image to be detected based on the original image and social transmission media. Or

[0127] In this way, the original image is obtained after being transmitted through the actual social transmission media. That is, when the social transmission media transmits the original image, it will inevitably add transmission noise to the original image. In this way, the image obtained again is the image to be detected carrying transmission noise.

[0128] The above method obtains the image to be detected based on the original image and the social transmission medium, see Fig. 9 As shown, including:

[0129] Step 2011: Upload the original image to a social transmission medium for display, so that the social transmission medium adds transmission noise to the original image.

[0130] During the implementation process, an upload tool can be pre-set to achieve batch uploading of multiple original images. After the original images are uploaded to the social transmission media for display, due to the inevitable existence of noise in the social transmission media, the social transmission media will add transmission noise to the original images accordingly.

[0131] Step 2012: Download the original image after adding transmission noise from the social transmission media, and use the original image after adding transmission noise as the image to be detected.

[0132] After the social transmission media adds transmission noise to the original image, that is, after the original image is displayed on the social transmission media, the original image with added transmission noise can be downloaded from the social transmission media through a pre-set download tool. In this case, the original image with added transmission noise is used as the image to be detected. Compared with the original image, the transmission noise carried by the image to be detected in this way cannot be ignored.

[0133] Method 2: Obtain the image to be detected based on the original image and the pre-trained transmission noise model.

[0134] During implementation, the image to be detected can also be obtained through a pre-trained transmission noise model.

[0135] Specifically, the image to be detected is obtained based on the original image and the pre-trained transmission noise model, see Fig.10 As shown, including:

[0136] Step 2011': input the original image into a pre-trained transmission noise model for learning, so that the transmission noise model outputs the original image carrying simulated transmission noise, wherein the simulated transmission noise is obtained from the output of the original transmission noise model when it is trained as the transmission noise model.

[0137] Since the original transmission noise model will obtain simulated transmission noise in the output when it is trained as a transmission noise model, based on this, during the implementation process, the above original image is input into the pre-trained transmission noise model for learning. During the learning process, the transmission noise model will add the simulated transmission noise to the original image. In this way, the transmission noise model will output the original image carrying the simulated transmission noise.

[0138] Step 2012 ′: taking the original image carrying the simulated transmission noise as the image to be detected.

[0139] During the implementation, the original image carrying the simulated transmission noise is used as the image to be detected corresponding to the original image. Similarly, compared with the original image, the transmission noise carried by the image to be detected cannot be ignored.

[0140] In addition, considering the camera characteristics determined by the hardware characteristics of the camera, in the embodiment of the present application, while determining the image to be detected, it is also necessary to obtain a camera fingerprint that characterizes the above-mentioned camera characteristics.

[0141] Get the camera fingerprint corresponding to the image to be detected, see Fig.11 As shown, including:

[0142] Step 2013: input the image to be detected into a pre-trained camera fingerprint extraction model for detection, and obtain the camera fingerprint corresponding to the image to be detected.

[0143] Since the camera fingerprint extraction model is trained based on the original camera fingerprint extraction model, during the implementation process, after the image to be detected is input into the pre-trained camera fingerprint extraction model for detection, the camera fingerprint corresponding to the above image to be detected can be output. It should be noted that the camera fingerprint is an indicator of the camera feature and is a useful criterion for identifying whether an image has been tampered with.

[0144] Correspondingly, the camera fingerprint of the image to be detected should be the same as the above camera fingerprint when it has not been tampered with. Once the image to be detected has been tampered with, the corresponding camera fingerprint may have been tampered with.

[0145] Step 202: Input the image to be detected and the camera fingerprint into a pre-trained tampering detection model for detection, and obtain the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected.

[0146] For the input image to be detected and camera fingerprint, after using the pre-trained tampering detection model for detection, the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected can be obtained. It should be noted that the above tampering traces may be obvious or not obvious in the image to be detected. Fig.12 As shown, including:

[0147] Step 221: Input the image to be detected and the camera fingerprint into a pre-trained tampering detection model, and use the encoder of the tampering detection model to encode the image to be detected and the camera fingerprint respectively to obtain a first encoding feature corresponding to the image to be detected and a second encoding feature corresponding to the camera fingerprint.

[0148] Depend on Figure 6 It can be seen that the tampering detection model includes two encoders, but the inputs of the above two encoders share a weight, that is, the parameter performance of the above two encoders is the same, and they are the same encoders. After the image to be detected and the camera fingerprint are input into the pre-trained tampering detection model, the above two identical encoders are used to encode the image to be detected and the camera fingerprint respectively, so as to obtain the first encoding feature corresponding to the image to be detected and the second encoding feature corresponding to the camera fingerprint. During the implementation process, the tampering detection model may also include an encoder, and the above image to be detected and the camera fingerprint can be successively input into the encoder for encoding, that is, through two parallel processing processes, the first encoding feature and the second encoding feature are obtained respectively.

[0149] It should be noted that before the encoder using the tampering detection model encodes the image to be detected and the camera fingerprint respectively, refer to Fig.13 As shown, it also includes:

[0150] Step 2020: Use the tampering detection model to pre-convolute the camera fingerprint so that the dimension of the obtained camera fingerprint is the same as the dimension of the image to be detected.

[0151] Since the dimensions of the image to be detected and the camera fingerprint are different, during the implementation process, before using the encoder for encoding, the tampering detection model is used to pre-convolute the camera fingerprint. For example, the camera fingerprint is pre-convolved with 3*3*3, so that the dimension of the obtained camera fingerprint is the same as the dimension of the image to be detected.

[0152] Step 2022: Use the cross-fusion module of the tampering detection model to fuse the first coding feature and the second coding feature to obtain a fused feature.

[0153] During the implementation process, after obtaining the first coding feature and the second coding feature, the first coding feature and the second coding feature are further input into the cross-fusion module of the tampering detection model, and then the fusion feature is obtained according to the first coding feature and the second coding feature.

[0154] Preferably, the cross fusion module of the tampering detection model is used to fuse the first coding feature and the second coding feature to obtain a fusion feature, see Fig.14 As shown, including:

[0155] Step 2022': Use the cross-fusion module in the tampering detection model to perform cross-convolution fusion on the first encoding feature and the second encoding feature to obtain a fusion feature.

[0156] In the embodiment of the present application, the dimensions of the first coding feature and the second coding feature input into the cross-fusion module in the tampering detection model are the same. Figure 7 It can be seen that the above-mentioned cross-fusion module includes a cross-attention unit, which can perform cross-attention convolution fusion on the first coding feature and the second coding feature while transposing, matrix multiplying, etc. the first coding feature and the second coding feature respectively, thereby obtaining a fusion feature based on the first coding feature and the second coding feature.

[0157] Step 2023: Use the decoder of the tampering detection model to decode the fused features to obtain the pixel area corresponding to the pixel unit corresponding to the tampering trace in the image to be detected.

[0158] Combination Figure 6It can be seen that after using the cross-fusion module to obtain the fusion features, in order to determine the pixel-level position of the tampering traces in the image to be detected, during the implementation process, the decoder of the tampering detection model is used to decode the above fusion features. It should be supplemented that the decoding process is opposite to the above encoding process, thereby converting the fusion features into image expressions, and then determining the pixel area corresponding to the tampering traces in the image to be detected.

[0159] Step 2024: Use the classification head of the tampering detection model to extract fusion features, and obtain the tampering probability of the image to be detected based on the fusion features.

[0160] Considering the correlation between the tampering location task and the tampering detection task, in the embodiment of the present application, after determining the pixel area corresponding to the tampering trace in the image to be detected, it is necessary to further determine the tampering probability of the image to be detected. During the implementation process, the classification head of the tampering detection model is used to extract the fusion feature, that is, the fusion feature is input into the classification head for classification prediction. The classification head contains two fully connected layers. The number of output nodes of the first fully connected layer is 512, and the number of output nodes of the second fully connected layer is 2, so that the tampering probability of the image to be detected can be determined according to the extraction result of the classification head, thereby improving the tampering detection result.

[0161] Based on the same inventive concept, refer to Fig.15 As shown, an embodiment of the present application provides a device for detecting tampering of a digital image, comprising:

[0162] An acquisition unit 1501 is used to acquire a camera fingerprint of an image to be detected, wherein the image to be detected is an image carrying traces of tampering, and the camera fingerprint is determined based on a pre-trained camera fingerprint extraction model, and the camera fingerprint is used to characterize the camera characteristics of the camera of the image to be detected;

[0163] The detection unit 1502 is used to input the image to be detected and the camera fingerprint into a pre-trained tampering detection model for detection, and obtain the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected.

[0164] Optionally, the image to be detected is obtained by:

[0165] Acquire the image to be detected based on the original image; or

[0166] Acquire the image to be detected based on the original image and the social transmission medium; or

[0167] Acquire the image to be detected based on the original image and the pre-trained transmission noise model;

[0168] Among them, the original image is taken by a camera with camera fingerprint.

[0169] Optionally, the image to be detected is acquired based on the original image and the social transmission medium, and the acquisition unit 1501 is used to:

[0170] Uploading the original image to a social transmission medium for display, so that the social transmission medium adds transmission noise to the original image;

[0171] The original image after adding transmission noise is downloaded from the social transmission media, and the original image after adding transmission noise is used as the image to be detected.

[0172] Optionally, the image to be detected is acquired based on the original image and a pre-trained transmission noise model, and the acquisition unit 1501 is used to:

[0173] Inputting the original image into a pre-trained transmission noise model for learning, so that the transmission noise model outputs the original image with simulated transmission noise, wherein the simulated transmission noise is obtained from the output of the original transmission noise model trained as the transmission noise model;

[0174] The original image with simulated transmission noise is used as the image to be detected.

[0175] Optionally, to obtain a camera fingerprint of an image to be detected, the obtaining unit 1501 is used to:

[0176] The image to be detected is input into the pre-trained camera fingerprint extraction model for detection to obtain the camera fingerprint corresponding to the image to be detected.

[0177] Optionally, the image to be detected and the camera fingerprint are input into a pre-trained tampering detection model for detection, and the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected are obtained. The detection unit 1502 is used to:

[0178] Input the image to be detected and the camera fingerprint into the pre-trained tampering detection model, and use the encoder of the tampering detection model to encode the image to be detected and the camera fingerprint respectively to obtain a first encoding feature corresponding to the image to be detected and a second encoding feature corresponding to the camera fingerprint;

[0179] Using the cross fusion module of the tampering detection model to fuse the first coding feature and the second coding feature to obtain a fusion feature;

[0180] The decoder of the tampering detection model is used to decode the fused features to obtain the pixel area corresponding to the pixel unit corresponding to the tampering trace in the image to be detected;

[0181] The classification head of the tampering detection model is used to extract fusion features, and the tampering probability of the image to be detected is obtained based on the fusion features.

[0182] Optionally, before the encoder of the tampering detection model encodes the image to be detected and the camera fingerprint respectively, it also includes:

[0183] Use the tampering detection model to pre-convolute the camera fingerprint so that the dimension of the obtained camera fingerprint is the same as the dimension of the image to be detected;

[0184] The cross fusion module of the tampering detection model is used to fuse the first coding feature and the second coding feature to obtain a fusion feature, including:

[0185] The cross-fusion module in the tampering detection model is used to perform cross-convolution fusion on the first encoding feature and the second encoding feature to obtain a fused feature.

[0186] Based on the same inventive concept, refer to Fig.16 As shown, an embodiment of the present application provides a computing device, including: a memory 1601, used to store executable instructions; a processor 1602, used to read and execute the executable instructions stored in the memory, and execute any one of the methods of the first aspect above.

[0187] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute the method described in any one of the first aspects above.

[0188] In summary, in an embodiment of the present application, a method, device and storage medium for tampering detection of a digital image are provided, the method comprising: obtaining a camera fingerprint of an image to be detected, wherein the image to be detected is an image carrying traces of tampering, the camera fingerprint is determined based on a pre-trained camera fingerprint extraction model, the camera fingerprint is used to characterize the camera characteristics of the camera of the image to be detected, the image to be detected and the camera fingerprint are input into a pre-trained tampering detection model for detection, and the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected are obtained. In this solution, the tampering detection model is used to accurately detect the pixel areas where various types of tampering traces are located in the image to be detected and obtain the tampering probability of the image to be detected, thereby effectively improving the accuracy of tampering detection.

[0189] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program product systems. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product system implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0190] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program product systems according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0193] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting tampering of a digital image, characterized in that: The method comprises: Obtaining a camera fingerprint of an image to be detected, wherein the image to be detected is an image carrying traces of tampering, and the camera fingerprint is determined based on a pre-trained camera fingerprint extraction model, and the camera fingerprint is used to characterize the camera characteristics of the camera of the image to be detected; The image to be detected and the camera fingerprint are input into a pre-trained tampering detection model for detection to obtain the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected.

2. The method according to claim 1, characterized in that The image to be detected is obtained by the following method: Acquire the image to be detected based on the original image; or Acquire the image to be detected based on the original image and the social transmission medium; or Acquire the image to be detected based on the original image and the pre-trained transmission noise model; The original image is taken by a camera having the camera fingerprint.

3. The method according to claim 2, characterized in that The step of acquiring the image to be detected based on the original image and the social transmission medium includes: Uploading the original image to the social transmission medium for display, so that the social transmission medium adds transmission noise to the original image; The original image after adding the transmission noise is downloaded from the social transmission medium, and the original image after adding the transmission noise is used as the image to be detected.

4. The method according to claim 3, characterized in that The step of acquiring the image to be detected based on the original image and the pre-trained transmission noise model includes: Inputting the original image into a pre-trained transmission noise model for learning, so that the transmission noise model outputs the original image carrying simulated transmission noise, wherein the simulated transmission noise is obtained from the output of the original transmission noise model trained as the transmission noise model; The original image carrying the simulated transmission noise is used as the image to be detected.

5. The method according to claim 1, characterized in that The step of obtaining the camera fingerprint of the image to be detected includes: The image to be detected is input into the pre-trained camera fingerprint extraction model for detection to obtain the camera fingerprint corresponding to the image to be detected.

6. The method according to claim 1, characterized in that The step of inputting the image to be detected and the camera fingerprint into a pre-trained tampering detection model for detection, and obtaining a pixel area corresponding to the tampering trace in the image to be detected and a tampering probability of the image to be detected, includes: Input the image to be detected and the camera fingerprint into a pre-trained tampering detection model, and use the encoder of the tampering detection model to encode the image to be detected and the camera fingerprint respectively to obtain a first encoding feature corresponding to the image to be detected and a second encoding feature corresponding to the camera fingerprint; Using the cross fusion module of the tampering detection model to fuse the first coding feature and the second coding feature to obtain a fused feature; Decoding the fused features using the decoder of the tampering detection model to obtain a pixel area corresponding to the pixel unit corresponding to the tampering trace in the image to be detected; The classification head of the tampering detection model is used to extract the fusion feature, and the tampering probability of the image to be detected is obtained based on the fusion feature.

7. The method according to claim 6, characterized in that Before the encoder using the tamper detection model encodes the image to be detected and the camera fingerprint respectively, it also includes: Using the tampering detection model to pre-convolute the camera fingerprint, so that the dimension of the obtained camera fingerprint is the same as the dimension of the image to be detected; The cross-fusion module using the tampering detection model performs feature fusion on the first coding feature and the second coding feature to obtain a fusion feature, including: The first coding feature and the second coding feature are cross-convolved and fused using the cross-fusion module in the tampering detection model to obtain the fused feature.

8. A device for detecting tampering of a digital image, characterized in that: include: an acquisition unit, configured to acquire a camera fingerprint of an image to be detected, wherein the image to be detected is an image carrying traces of tampering, the camera fingerprint is determined based on a pre-trained camera fingerprint extraction model, and the camera fingerprint is used to characterize camera characteristics of a camera of the image to be detected; The detection unit is used to input the image to be detected and the camera fingerprint into a pre-trained tampering detection model for detection, and obtain the pixel area corresponding to the tampering trace in the image to be detected and the tampering probability of the image to be detected.

9. A computing device, characterized in that include: A memory for storing executable instructions; A processor, configured to read and execute the executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 7.

Images