An attention-based object semantic consistency tampered image detection method

Abstract

The application discloses a tampered image detection method based on object semantic consistency, and relates to the field of image passive forensics. Most of the existing splicing tampering detection methods based on deep learning are realized based on anomaly detection, and there are limitations in the receptive field due to convolution operation. In view of the problem, the application adopts an encoder constructed based on an attention mechanism to respectively extract and represent the semantic information of the object level and the tampering trace information of the pixel level in the image, generates object semantic representation and tampering trace representation, and fuses and decodes two different encodings through a decoder, so that splicing tampering detection and positioning of the image are realized. The application can combine object semantic consistency and suspicious tampering traces, effectively improve the detection accuracy of the tampered image, and has practical value.

Description

Technical Field

[0001] This invention relates to the field of image forensics, and in particular to an attention-based method for detecting tampered images with semantic consistency of objects. Background Technology

[0002] Images, as a medium for information exchange, are widely used across various industries. However, the prevalence of image editing technology has also made image tampering easy. Tampered images destroy the integrity of their content, and if used for illegal purposes, they can impact the stability of cyberspace security. Therefore, image tampering detection has become a pressing issue in multimedia forensics within the field of information security. Image tampering involves editing natural images; the image from which the tampered area originates is the donor image, and the image receiving the tampered area is the recipient image. Based on the consistency of the donor and recipient images, tampered images can be categorized into same-source tampered images and different-source tampered images. In same-source tampering, the donor and recipient images are the same image, such as in copy-move tampering. In different-source tampering, the donor and recipient images are not the same image, and there may be multiple donor images, such as in splicing tampering. The tampering process leaves inherent traces, such as discontinuities at the edges of the tampered area and inconsistencies between the tampered and untampered areas, which can be used to detect tampered images and locate the tampered area. In image splicing tampering, the original semantic information of the image often changes, and the objects of splicing tampering operations often have complete semantic information. However, existing deep learning-based image tampering detection methods cannot fully combine the semantic consistency of the object with suspicious tampering traces to extract information of the tampered image for detection within a larger receptive field. For example, in the papers "The point where reality meets fantasy: mixed adversarial generators for image splice detection" (Proceedings of the 33rd International Conference on Neural Information Processing Systems, 215-226) and "Semantic Consistency Guided Multi-Task Splicing Tampering Detection Algorithm" (Journal of Image and Graphics, 2023, 28(3)), anomaly detection is based on convolutional operations. Their receptive fields are limited, and they all require additional semantic labels to assist in obtaining semantic information, which is greatly limited by the dataset. Summary of the Invention

[0003] The purpose of this invention is to overcome the above-mentioned limitations and provide an attention-based object semantic consistency tampering image detection method to further improve the accuracy of identification and detection of tampered regions.

[0004] The technical solution for achieving the objective of this invention is as follows:

[0005] An attention-based method for detecting object semantic consistency tampering in images employs an encoder built on an attention mechanism to extract and represent features of semantic information at the object level and tampering trace information at the pixel level, generating object semantic representation and tampering trace representation. A decoder then fuses and decodes these two different encodings to achieve image splicing tampering detection and localization. The method includes the following steps:

[0006] Step 1: Construct an object semantic encoder based on a self-attention mechanism, which includes a pre-convolutional module, eight stacked object semantic encoding modules, and a multilayer perceptron.

[0007] Step 2: Construct an attention-based tamper trace encoder to obtain tamper traces of color components and high-frequency components in the image.

[0008] Step 3: Construct a fusion decoder to locate the tampered area by combining the object semantic information and tampering trace information obtained in Step 1 and Step 2.

[0009] Step 4: During the training phase, the loss function is used to provide overall constraints for the algorithm obtained in steps 1 to 3, so that the parameters in the algorithm are adjusted to the optimal values.

[0010] Furthermore, in step 1, the specific method for implementing object semantic encoding is as follows:

[0011] (1) Use the pre-convolution module to process the image I to be detected, which has a length of H, a width of W, and a dimension of C. H×W Perform shallow feature extraction, pre

[0012] Let the sliding window size be k×k, the sliding step size be s, and the fill size be p. The process can be expressed as:

[0013] OSB i =cut(Conv(I H×W ))

[0014] Where Conv(·) represents the convolution operation of the preceding convolution module, and cut(·) represents the slicing operation;

[0015] (2) The object semantic encoding module first uses position encoding to encode the OSB i The process of updating and generating the updated sequence block is represented as follows:

[0016]

[0017] Where embedding(·) represents position encoding;

[0018] (3) The object-level semantic consistency relationship between image patch sequences is modeled by a multi-head self-attention layer. The calculation process is expressed as follows:

[0019]

[0020]

[0021] in, For the initial object semantic representation model, d head The feature scaling factor is adjustable; in this method, d is generally used. head =64, softmax(·) represents the softmax function; K e Q e and V e All are outputs of learnable linear projection layers. The linear mapping, i.e., the query, key, and value vector computed by the attention mechanism, has a dimension that is... Consistent, all are Here, N=16 is an adjustable parameter, representing the maximum number of semantic objects that an image may contain, and C is... The number of channels;

[0022] (4) A multilayer perceptron is used to obtain the semantic relationships between different objects in the same image, and to learn how different object prototypes interact. Finally, the multilayer perceptron and activation functions are used to enhance the object semantic model obtained by the module to achieve better generalization. This process is represented as follows:

[0023]

[0024]

[0025] Among them, W C These are learnable parameters in a multilayer perceptron; among them, OS i This is the object semantic representation obtained by the object semantic encoder. GELU(·) represents the GELU activation function, and MLP(·) represents the linear mapping operation of the multilayer perceptron.

[0026] Furthermore, in step 2, the specific method for extracting tampering traces is as follows:

[0027] (1) Extract the tampering traces corresponding to the color components, representing FC. RGB ;

[0028] (2) Extracting tampering traces corresponding to high-frequency components to represent FC HF ;

[0029] (3) FCRGB With FC HF Adding them along the channel dimension yields image I. H×W Corresponding tampering feature code

[0030] (4) The sampling layer further processes the tampered feature encoding, scaling and slicing it into blocks, ultimately generating a result consistent with the OS. i Tampering features with the same dimension represent FC i .

[0031] Furthermore, in step 3, the specific method for feature fusion and decoding by the fusion decoder is as follows:

[0032] (1) OS i and FC i The data is then fed into a multi-head self-attention layer and a multilayer perceptron for block embedding refinement to obtain the refined representation. The calculation process is as follows:

[0033]

[0034]

[0035] Among them, IV i Q is an intermediate variable in the calculation process. d For FC i linear mapping, K d and V d All are OS i A linear mapping, Q d K d K d The dimensions are all the same as FC i OS i Consistent, all are

[0036] (2) Construct a BCIM layer Reshape it into a new dimension H R ×W R ×2C R Two-dimensional feature map P i Then, fine-grained feature modeling between blocks is performed by leveraging boundary-sensitive contextual inconsistencies in the block sequence; specifically, feature map P is computed using a k×k sliding window. i The cosine similarity between each pixel and its surrounding pixels can be represented as:

[0037]

[0038] Among them, P ijP represents the center feature vector of a sliding window of size k×k. ik This indicates that within a sliding window of size k×k, the relationship with P... ij The neighboring feature vectors of adjacent regions are used, and CosSim(·) represents the cosine similarity calculation function; then, based on the cosine similarity S between each pixel... ij Get P i The corresponding overall similarity is represented by S. i Its dimension is H R ×W R ×1;

[0039] (3) S obtained from multiple different decoding layers i and P i Element-wise concatenation and serialization yield a block sequence that integrates object semantic features and tampering traces. This block sequence is then processed by an upsampling layer to obtain the predicted tampering detection result D. out .

[0040] Furthermore, in step 4, the loss function used by the network is:

[0041] LOSS = BCE(GT, D) out )

[0042] Where BCE(·) represents the cross-entropy loss function, GT represents the actual tampered region in the image, and D out This indicates the actual tampering detection result output by the network.

[0043] The method of this invention can detect tampered images by starting from object-level semantic consistency and combining tampering traces in image color components and high-frequency components, and use deep learning technology to ensure image content security.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. This invention utilizes a self-attention mechanism, which enables the network to focus on the correlation between pixels that are far apart, thereby expanding the network's receptive field.

[0046] 2. Object-level semantic consistency information enables the network to simulate the thinking process of the human brain, perceive anomalies in forged images at the semantic level, and achieve more accurate tamper detection.

[0047] 3. By combining object-level semantic consistency information with tampering trace information in color components and high-frequency components, the decoder can better locate tampered areas. Attached Figure Description

[0048] Figure 1 This is a flowchart of an embodiment of the present invention.

[0049] Figure 2 This is a network structure diagram of an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of an object semantic encoder according to an embodiment of the present invention.

[0051] Figure 4 This is a schematic diagram of a tamper trace encoder according to an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram of a fusion decoder according to an embodiment of the present invention.

[0053] Figure 6 This is a list of detection results from an embodiment of the present invention.

[0054] Figure 7 This is a comparison graph of the experimental results of the present invention with existing methods in the publicly available verification examples. Detailed Implementation

[0055] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0056] like Figures 1-2 As shown, this embodiment of the invention constructs an attention-based object semantic consistency tampering image detection network, which includes an object semantic encoder, a tampering trace encoder, and a fusion decoder, forming the entire algorithm framework. Figure 1 The workflow of this invention is described; Figure 2 A specific network structure diagram of the present invention under various embodiments is provided.

[0057] This invention discloses an attention-based method for detecting object semantic consistency tampering images, comprising the following main steps:

[0058] Step 1: Construct an object semantic encoder based on a self-attention mechanism, which includes a pre-convolutional module, eight stacked object semantic encoding modules, and a multilayer perceptron.

[0059] The specific steps of object semantic encoding are as follows: 1) Use a pre-convolutional module to encode the image I to be detected, which has a length of H, a width of W, and a dimension of C. H×W Shallow feature extraction is performed, with a preset sliding window size of k×k, a sliding step size of s, and a fill size of p. This process can be represented as:

[0060] OSB i =cut(Conv(I H×W ))

[0061] Here, Conv(·) represents the convolution operation of the preceding convolution module, and cut(·) represents the slicing operation.

[0062] 2) The object semantic encoding module first uses positional encoding to encode the OSB. i The process of updating and generating the updated sequence block can be represented as follows:

[0063]

[0064] Here, embedding(·) represents position encoding.

[0065] 3) A multi-head self-attention layer models the object-level semantic consistency relationships between image patch sequences. This calculation process can be represented as:

[0066]

[0067]

[0068] in, For the initial object semantic representation model, d head =64 is an adjustable feature scaling factor, and softmax(·) represents the softmax function. K e Q e and V e All are outputs of learnable linear projection layers. The linear mapping, i.e., the query, key, and value vector computed by the attention mechanism, has a dimension that is... Consistent, all are Here, N=16 is an adjustable parameter, representing the maximum number of semantic objects that an image may contain.

[40] C is The number of channels.

[0069] 4) A multilayer perceptron is used to acquire the semantic relationships between different objects in the same image, learning how different object prototypes interact. Finally, the multilayer perceptron and activation functions are used to enhance the object semantic model obtained by the module to achieve better generalization. This process can be represented as:

[0070]

[0071]

[0072] Among them, W C These are learnable parameters in a multilayer perceptron. Among them, OS... i This is the object semantic representation obtained by the object semantic encoder. GELU(·) represents the GELU activation function, and MLP(·) represents the linear mapping operation of the multilayer perceptron.

[0073] Step 2: Construct an attention-based tamper trace encoder to extract and encode tamper traces contained in the color components and high-frequency components of the forged image, respectively. The tamper trace encoder generates color component feature codes FC in the forged image. RGB Feature encoding of high-frequency components FC HF Finally, further processing yields the tampering feature representation (FC) of the forged image. i .

[0074] Step 3: Construct a fusion decoder. Obtain the semantic objects contained in the image by using the object-level information provided by the object semantic encoder. Combine the tamper trace encoding provided by the tamper trace encoder to further identify pixels with traces and semantic objects with suspicious boundaries in the image. Finally, combine the features of the two levels to output the predicted tampered region.

[0075] Step 4: Provide overall constraints to the algorithm using a loss function. The binary cross-entropy loss (BCE Loss) function is used to add overall constraints to the network. The calculation process is as follows:

[0076] LOSS = BCE(GT, D) out )

[0077] Where BCE(·) represents the cross-entropy loss function, GT represents the actual tampered region in the image, and D out This indicates the actual tampering detection result output by the network.

[0078] Example:

[0079] This embodiment includes the following main steps:

[0080] S1: Construct an object semantic encoder based on a self-attention mechanism, which includes a pre-convolutional module, eight stacked object semantic encoding modules, and a multilayer perceptron.

[0081] like Figure 3 As shown, the specific steps of object semantic encoding are as follows:

[0082] S1.1: Use a pre-convolutional module to process the image I to be detected, which has a length of H, a width of W, and a dimension of C. H×W Shallow feature extraction is performed, with a preset sliding window size of k×k, a sliding step size of s, and a fill size of p. This process can be represented as:

[0083] OSB i =cut(Conv(I H×W ))

[0084] Where Conv(·) represents the convolution operation of the preceding convolution module, and cut(·) represents the slicing operation. The output of step S1.1 will be used as the input of S1.2.

[0085] S1.2: The object semantic encoding module first uses positional encoding to pair the OSB i The process of updating and generating the updated sequence block can be represented as follows:

[0086]

[0087] Here, embedding(·) represents position encoding.

[0088] S1.3: A multi-head self-attention layer models the object-level semantic consistency relationships between image patch sequences. This calculation process can be represented as:

[0089]

[0090]

[0091] in, For the initial object semantic representation model, d head =64 is an adjustable feature scaling factor, and softmax(·) represents the softmax function. K e Q e and V e All are outputs of learnable linear projection layers. The linear mapping, i.e., the query, key, and value vector computed by the attention mechanism, has a dimension that is... Consistent, all are Here, N=16 is an adjustable parameter, representing the maximum number of semantic objects that an image may contain.

[40] C is The number of channels.

[0092] S1.4: A multilayer perceptron is used to acquire the semantic relationships between different objects in the same image, learning how different object prototypes interact. Finally, the multilayer perceptron and activation functions are used to enhance the object semantic model obtained by the module to achieve better generalization. This process can be represented as:

[0093]

[0094]

[0095] Among them, W c These are learnable parameters in a multilayer perceptron. Among them, OS... iThis is the object semantic representation obtained by the object semantic encoder. GELU(·) represents the GELU activation function, and MLP(·) represents the linear mapping operation of the multilayer perceptron.

[0096] S2: Construct an attention-based tamper trace encoder to extract and encode tamper traces contained in the color components and high-frequency components of the forged image, respectively. The parameters of the convolution kernel used in the high-frequency component extraction layer are:

[0097]

[0098] like Figure 4 As shown, the tampering trace coding process is as follows:

[0099] S2.1: Image I to be detected H×W The input is fed into the RGB channels, and features are extracted through the residual attention unit to generate the detection image I. H×W The tampering traces corresponding to the color components are represented by FC. RGB .

[0100] S2.2: In the SRM channel, the high-frequency component extraction layer first extracts the image I to be detected. H×W The process is performed, and the result is used as input for S2.3.

[0101] S2.3: The image to be detected after being processed by the high-frequency component extraction layer. H×W Feature extraction is performed using residual attention units to generate tamper trace representations (FCs) corresponding to their high-frequency components. HF .

[0102] S2.4: FC RGB With FC HF Adding them along the channel dimension yields image I. H×W Corresponding tampering feature code

[0103] S2.5: Sampling layer for encoding tampered features Further processing involves scaling and slicing the data to ultimately generate a result compatible with the OS. i Tampering features with the same dimension represent FC i .

[0104] S3: Construct a fusion decoder to obtain semantic objects contained in the image through the object-level information provided by the object semantic encoder, and further identify pixels with traces and semantic objects with suspicious boundaries in the image by combining the tamper trace encoding provided by the tamper trace encoder. Finally, combine the features of the two levels to output the predicted tampered region.

[0105] like Figure 5 As shown, the specific steps of the fusion decoder operation are as follows:

[0106] S3.1: Transfer the OS from steps S1 and S2 i and FC i The data is then fed into a multi-head self-attention layer and a multilayer perceptron for block embedding refinement to obtain the refined representation. The calculation process is as follows:

[0107]

[0108]

[0109] Among them, IV i Q is an intermediate variable in the calculation process. d For FC i linear mapping, K d and V d All are OS i A linear mapping, Q d K d K d The dimensions are all the same as FC i OS i Consistent, all are

[0110] S3.2: Construct a BCIM layer Reshape it into a new dimension H R ×W R ×2C R Two-dimensional feature map P i Then, fine-grained feature modeling between blocks is performed by leveraging boundary-sensitive contextual inconsistencies within the block sequence. Specifically, this involves calculating the feature map P using a k×k sliding window. i The cosine similarity between each pixel and its surrounding pixels can be represented as:

[0111]

[0112] Among them, P ij P represents the center feature vector of a sliding window of size k×k. ik This indicates that within a sliding window of size k×k, the relationship with P... ij The neighboring feature vectors of adjacent regions are represented by CosSim(·), which denotes the cosine similarity calculation function. Then, based on the cosine similarity S between each pixel... ij Get P i The corresponding overall similarity is represented by S. i Its dimension is H R ×W R ×1.

[0113] S3.3: S obtained from multiple different decoding layers i and P i Element-wise concatenation and serialization yield a block sequence that integrates object semantic features and tampering traces. This block sequence is then processed by a sampling layer to obtain the predicted tampering detection result D. out .

[0114] S4: Provide overall constraints to the algorithm through the loss function. The binary cross-entropy loss (BCE Loss) function is used to add overall constraints to the network. The calculation process is as follows:

[0115] LOSS = BCE(GT, D) out )

[0116] Where BCE(·) represents the cross-entropy loss function, GT represents the actual tampered region in the image, and D out This indicates the actual tampering detection result output by the network.

[0117] This embodiment uses precision as the evaluation metric: Wherein, TP represents the number of pixels in the correctly detected stitched area, and FP represents the total number of pixels in the detection result that were incorrectly detected as tampered areas.

[0118] Figure 6 This paper compares the experimental results of the proposed method with existing methods in a publicly available verification example. The results show that the proposed method performs well on both the Fantastic Reality and Spliced ​​Dataset datasets. It demonstrates superior detection performance compared to the contrasting algorithms.

[0119] Figure 7 The diagram shows the detection results of a verification example of the present invention, specifically illustrating the detection and segmentation effects of the method proposed in this invention.

Claims

1. An attention-based method for detecting object semantic consistency tampering in images, employing an encoder built on an attention mechanism to extract and represent features of semantic information at the object level and tampering trace information at the pixel level, generating object semantic representation and tampering trace representation. A decoder then fuses and decodes these two different encodings to achieve image splicing tampering detection and localization. The method includes the following steps: Step 1: Construct an object semantic encoder based on a self-attention mechanism, which includes a pre-convolutional module, eight stacked object semantic encoding modules, and a multilayer perceptron; Step 2: Construct an attention-based tamper trace encoder to obtain tamper traces of color components and high-frequency components in the image; Step 3: Construct a fusion decoder to locate the tampered area by combining the object semantic information and tampering trace information obtained in Step 1 and Step 2; Step 4: During the training phase, the loss function provides overall constraints for the algorithm obtained in Steps 1 to 3, so that the parameters in the algorithm are adjusted to optimal values; In step 1, the specific method for implementing object semantic encoding is as follows: (1) Use the pre-convolution module to process the image I to be detected, which has a length of H, a width of W, and a dimension of C. H×W Shallow feature extraction is performed, with a preset sliding window size of k×k, a sliding step size of s, and a fill size of p. This process is represented as: OSB i =cut(Conv(I H×W )) Where Conv(·) represents the convolution operation of the preceding convolution module, and cut(·) represents the slicing operation; (2) The object semantic encoding module first uses position encoding to encode the OSB i The process of updating and generating the updated sequence block is represented as follows: Where embedding(·) represents position encoding; (3) The object-level semantic consistency relationship between image patch sequences is modeled by a multi-head self-attention layer. The calculation process is expressed as follows: in, For the initial object semantic representation model, d head The feature scaling factor is adjustable; in this method, d is generally used. head =64, softmax(·) represents the softmax function; K e Q e and V e All are outputs of learnable linear projection layers. The linear mapping, i.e., the query, key, and value vector computed by the attention mechanism, has a dimension that is... Consistent, all are Here, N is an adjustable parameter; in this method, N = 16, representing the maximum number of semantic objects that an image may contain. C is... The number of channels; (4) A multilayer perceptron is used to obtain the semantic relationships between different objects in the same image, and to learn how different object prototypes interact. Finally, the multilayer perceptron and activation functions are used to enhance the object semantic model obtained by the module to achieve better generalization. This process is represented as follows: Among them, W C These are learnable parameters in a multilayer perceptron; among them, OS i This is the object semantic representation obtained by the object semantic encoder. GELU(·) represents the GELU activation function, and MLP(·) represents the linear mapping operation of the multilayer perceptron.

2. The attention-based object semantic consistency tampering image detection method as described in claim 1, characterized in that, In step 2, the specific method for extracting tampering traces is as follows: (1) Extract the tampering traces corresponding to the color components, representing FC. RGB ; (2) Extracting tampering traces corresponding to high-frequency components to represent FC HF ; (3) FC RGB With FC HF Adding them along the channel dimension yields image I. H×W Corresponding tampering feature code (4) The sampling layer further processes the tampered feature encoding, scaling and slicing it into blocks, ultimately generating a result consistent with the OS. i Tampering features with the same dimension represent FC i .

3. The attention-based object semantic consistency tampering image detection method as described in claim 1, characterized in that, In step 3, the specific method for feature fusion and decoding by the fusion decoder is as follows: (1) OS i and FC i The data is then fed into a multi-head self-attention layer and a multilayer perceptron for block embedding refinement to obtain the refined representation. The calculation process is as follows: Among them, IV i Q is an intermediate variable in the calculation process. d For FC i linear mapping, K d and V d All are OS i A linear mapping, Q d K d K d The dimensions are all the same as FC i OS i Consistent, all are (2) Construct a BCIM layer Reshape it into a new dimension H R ×W R ×2C R Two-dimensional feature map P i Then, fine-grained feature modeling between blocks is performed by leveraging boundary-sensitive contextual inconsistencies in the block sequence; specifically, feature map P is computed using a k×k sliding window. i The cosine similarity between each pixel and its surrounding pixels can be represented as: Among them, P ij P represents the center feature vector of a sliding window of size k×k. ik This indicates that within a sliding window of size k×k, the relationship with P... ij The neighboring feature vectors of adjacent regions are used, and CosSim(·) represents the cosine similarity calculation function; then, based on the cosine similarity S between each pixel... ij Get P i The corresponding overall similarity is represented by S. i Its dimension is H R ×W R ×1; (3) S obtained from multiple different decoding layers i and P i Element-wise concatenation and serialization yield a block sequence that integrates object semantic features and tampering traces. This block sequence is then processed by an upsampling layer to obtain the predicted tampering detection result D. out .

4. The attention-based object semantic consistency tampering image detection method as described in claim 1, characterized in that, In step 4, the loss function used by the network is: LOSS=BCE(GT,D out ) Where BCE(·) represents the cross-entropy loss function, GT represents the actual tampered region in the image, and D out This indicates the actual tampering detection result output by the network.

5. A method for detecting object semantic consistency tampering based on attention as described in claim 1 or 3, characterized in that, In step 2, the specific parameters of the convolutional layer used in the attention-based tamper trace encoder high-frequency component extraction module are as follows:

Images