Real Scene Image Tampering Localization System and Method Based on Self-Attention Mechanism

Abstract

The present invention discloses a real-scene image forgery localization system and method based on a self-attention mechanism, which relates to the technical field of computer vision and includes a feature extraction module, a feature processing module, and a feature fusion module; the feature processing module includes N parallel feature processing branches; the real-scene forged image is input into the feature extraction module to obtain N initial feature maps at different stages; the Nth feature processing branch processes the Nth initial feature map to obtain the Nth attention feature matrix, which is input into the previous feature processing branch; for the first N-1 feature processing branches, each branch processes according to the corresponding initial feature map at the stage and combines the attention feature matrix of the subsequent branch to obtain the attention feature matrix of this branch, which is input into the previous branch; the feature fusion module performs feature fusion on the N attention feature matrices to obtain a real-scene image forgery mask map. The present invention can achieve accurate forgery localization in real-scene forged images.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and more specifically, to a real-scene image tampering localization system and method based on a self-attention mechanism. Background Art

[0002] Images are important carriers for disseminating information and are widely used in various social scenarios to convey important and sensitive information, such as commercial transactions, news reports, legal proceedings, etc. With the popularity of various image editing software, it has become very easy to tamper with images. The tampered images may be used for fraud, economic crimes or other illegal purposes, which are potential social hazards. The research on tampering forensics of these images that convey important information has certain social significance and helps to jointly build a security defense line for the cyber space and enhance the sense of security of the general public in the cyber space.

[0003] Due to the convenience of obtaining natural images, the research on the localization and detection of tampered images focuses on natural images. The research work carried out in multiple natural-scene tampered image datasets focuses on the image tampering features at the object scale of the scene. The tampering methods of natural-scene images are relatively single, but in real scenes, there are multiple categories of high-risk tampered images involving economic interests, usually forged qualification certificates, documents, screenshots, etc. As Figure 1 shown, the following are examples of two types of scene-tampered images and corresponding tampering masks. Figure 1 a) Schematic diagram of a natural-scene tampered image and the corresponding tampering mask. Figure 1 b) Schematic diagram of a real-scene certificate-type tampered image and the corresponding tampering mask. Figure 1 c) Schematic diagram of a real-scene document-type tampered image and the corresponding tampering mask. Figure 1 d) Schematic diagram of a real-scene screenshot-type tampered image and the corresponding tampering mask. The tampering areas of high-risk tampered images often concentrate on regions such as text, seals, barcodes or other regions that convey sensitive information, and usually undergo elaborate manual tampering interventions, with multiple post-processing methods and no fixed tampering pattern. Therefore, the tampering features of real-scene images may be very different from those of natural-scene images, and the performance of conventional natural-scene image tampering localization methods is limited in this scenario, and corresponding tampering localization methods need to be designed according to the image tampering characteristics in real scenes.

[0004] The prior art discloses a double-stream network forged image detection method based on local image patch feature extraction, including: segmenting the image to be detected into image patches; respectively inputting the obtained image patches into a CNN model and a CNN-GRU model to extract the internal artifacts of the image patches and the spatial features between the image patches, and respectively obtaining corresponding forgery scores. Among them, the closer the forgery score is to 1, the greater the possibility that the image to be detected is forged; using an attention-based fusion method to fuse the two obtained forgery scores to obtain a judgment score on the authenticity of the image to be detected. This method is not applicable to real-scene tampered images for face forged images; and this method can only finally output whether the image to be detected is forged, and cannot locate the forgery position. This application cannot achieve accurate tampering localization in real-scene tampered images. Summary of the Invention

[0005] In order to overcome the defect that the prior art has inaccurate tampering localization for real-scene tampered images, the present invention provides a real-scene image tampering localization system and method based on a self-attention mechanism, which can achieve accurate tampering localization in real-scene tampered images.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] The present invention provides a real-scene image tampering localization system based on a self-attention mechanism, including a feature extraction module, a feature processing module, and a feature fusion module; the feature processing module includes N parallel feature processing branches;

[0008] The real-scene tampered image is input into the feature extraction module for multi-stage feature extraction operations to obtain N-stage initial feature maps, and then input into the feature processing module; the Nth feature processing branch performs downsampling operations, flattening operations, and attention calculation operations on the Nth-stage initial feature map to obtain the Nth attention feature matrix, and then inputs it into the (N - 1)th feature processing branch and the feature fusion module; for the first N - 1 feature processing branches, each feature processing branch performs downsampling operations, flattening operations, addition operations with the attention feature matrix of the subsequent feature processing branch, and attention calculation operations on the corresponding stage initial feature map to obtain the attention feature matrix of this feature processing branch, and then inputs it into the previous feature processing branch and the feature fusion module; the feature fusion module performs feature fusion operations on the N attention feature matrices to obtain a real-scene image tampering mask map.

[0009] Preferably, the feature extraction module includes N sequentially connected convolutional neural network units;

[0010] Real-scene tampered image input feature extraction module. The first convolutional neural network unit extracts features from the real-scene tampered image to obtain the initial feature map f1 of the first stage, and inputs it to the input feature processing module and the second convolutional neural network unit. The second convolutional neural network unit extracts features from the initial feature map f1 of the first stage to obtain the initial feature map f2 of the second stage, and inputs it to the input feature processing module and the third convolutional neural network unit. This process continues until the Nth convolutional neural network unit extracts features from the initial feature map f of the N-1th stage N-1 to obtain the initial feature map f of the Nth stage N , and inputs it to the input feature processing module.

[0011] Preferably, the feature processing module includes N parallel feature processing branches. The Nth feature processing branch includes an overlapping convolutional unit, a flattening unit, and a full self-attention unit connected in sequence. The first N-1 feature processing branches have the same structure, each including an overlapping convolutional unit, a flattening unit, a first matrix addition point, and a full self-attention unit connected in sequence. The output end of the full self-attention unit in the jth feature processing branch is also connected to the matrix addition point in the j-1th feature processing branch, where j = 2, …, N;

[0012] The overlapping convolutional unit in each feature processing branch is correspondingly connected to the N convolutional neural network units in the feature extraction module, and each full self-attention unit is connected to the feature fusion module.

[0013] Preferably, the convolutional kernel size of the overlapping convolutional unit is 3×3.

[0014] The corresponding initial feature map f k undergoes two-fold downsampling through an overlapping convolutional layer with a convolutional kernel size of 3×3 and a stride of 2, and the number of channels is unified to the same dimension E to obtain the downsampled feature map. The flattening unit is used to flatten the corresponding downsampled feature map, restructuring the three-dimensional downsampled feature map into a two-dimensional matrix. The full self-attention unit in the Nth feature processing branch performs self-attention calculation on the corresponding two-dimensional matrix to obtain the Nth attention feature matrix, which is upsampled and then input to the matrix addition point in the N-1th feature processing branch, i.e., the inverse attention connection. In the first N-1 feature processing branches, the matrix addition point is used to add the two-dimensional matrix of this feature processing branch and the attention feature matrix of the subsequent feature processing branch, and then input it to the full self-attention unit of this feature processing branch for self-attention calculation to obtain the attention feature matrix of this feature processing branch.

[0015] Preferably, the specific method for the full self-attention unit to calculate the attention feature matrix is as follows:

[0016]

[0017] Where s k represents the k-th attention feature matrix, FAM k represents the k-th full self-attention unit, represents the k-th overlapping convolution unit, flatten k represents the k-th flattening unit, Up(*) represents the upsampling operation, f k represents the initial feature map of the k-th stage, k ∈ {1, 2, …, N}.

[0018] Preferably, the full self-attention unit includes P sequentially connected full self-attention networks; each full self-attention network includes a first normalization layer, a multi-head self-attention layer, a second matrix addition point, a second normalization layer, a hybrid multi-layer perceptron layer, and a third matrix addition point connected in sequence; the input end of the first normalization layer is also connected to the input end of the second matrix addition point, and the output end of the second matrix addition point is also connected to the input end of the third matrix addition point.

[0019] Preferably, the hybrid multi-layer perceptron layer includes a first fully connected layer, a depthwise separable convolution layer, a second fully connected layer, a global channel pooling layer, a third fully connected layer, a fourth fully connected layer, and a channel multiplication point connected in sequence; the output end of the second fully connected layer is also connected to the input end of the channel multiplication point.

[0020] Preferably, the feature fusion module includes a dimension rearrangement layer, a fifth fully connected layer, an upsampling layer, a channel splicing layer, a sixth fully connected layer, and a seventh fully connected layer connected in sequence.

[0021] The full self-attention unit can focus on tampering features from the global spatial scale, local spatial scale, global channel scale, and local channel scale; the full self-attention unit is composed of P sequentially connected full self-attention networks stacked, and each full self-attention network includes a first normalization layer, a multi-head self-attention layer, a second matrix addition point, a second normalization layer, a hybrid multi-layer perceptron layer, and a third matrix addition point connected in sequence; among them, the multi-head self-attention layer integrates multiple self-attention calculations, and the calculation formula is as follows:

[0022]

[0023] MHSA(Z) = [SA1(Z);...; SA m (Z)]W H

[0024]

[0025] Among them, Z represents the output of the first normalization layer, with the number of channels being E; the self-attention calculation self-learns and encodes Z into Q, K, and V matrices, and calculates the relationship between self-attention features in the self-learning encoding method; SA(*) represents the self-attention calculation, MHSA(*) represents the multi-head self-attention splicing, W H represents the parameter matrix, m represents the number of relationship heads, and d represents the number of dimensions;

[0026] The hybrid multi-layer perceptron layer further enriches the relationship information on the basis of the conventional multi-layer perceptron. It introduces depthwise separable convolution in the multi-layer perceptron to replace the fixed-position encoding in the attention mechanism. It can not only input images of any resolution, improving the applicability of the method, but also introduce local spatial domain information to a certain extent; the global channel pooling layer, the third fully connected layer, the fourth fully connected layer, and the channel multiplication point constitute the channel attention mechanism, which is used to focus on the possible tampering semantic information in the channels from the global scale. First, the global channel pooling layer and the ReLU function activation are used to obtain the global channel information. Then, the third fully connected layer and the fourth fully connected layer are connected and activated by the Sigmoid function to calculate the weight vector of each channel on the global scale. Finally, the weight vector and the input of the global channel pooling layer are multiplied and weighted by channels according to the importance of the channels on the global scale.

[0027] The full self-attention unit can focus on tampering features from the global spatial domain scale, local spatial domain scale, global channel scale, and local channel scale; for the full self-attention units in different feature processing branches, due to the different stages of the initial feature maps of the inputs in different feature processing branches, the image semantic information they focus on is different. The full self-attention units in the earlier feature processing branches tend to focus on the texture features in the real-scene tampered images, and the full self-attention units in the later feature processing branches tend to focus on the tampering traces; except for the last feature processing branch, the output of the full self-attention unit in the latter feature processing branch is connected to the input of the full self-attention unit in the previous feature processing branch by matrix addition, that is, inverse attention connection.

[0028] Preferably, the specific method for the feature fusion module to perform feature fusion operations on N attention feature matrices to obtain the real-scene image tampering mask map is as follows:

[0029] Input the N attention feature matrices into the feature fusion module. The attention feature matrix s k has a dimension of The attention feature matrix is reshaped from two-dimensional to three-dimensional by the dimension reshaping layer, and the dimension change is The attention feature matrix is unified to the same number of channels by the fifth fully connected layer, that is The size of the attention feature matrix is adjusted to the size of the attention feature matrix with the largest size among them, i.e., H×W×E′, through the upsampling layer; the attention feature matrices are concatenated in channels through the channel concatenation layer, i.e., H×W×NE′; the number of channels of the attention feature matrix is shrunk to 1 through the sixth fully connected layer and the seventh fully connected layer, i.e., H×W×1; finally, the attention feature matrix is activated and then decoded by the upsampling operation to obtain the probability map of the real scene image tampering mask:

[0030] M = Up(Sigmoid(SMM(S)))

[0031] where M represents the probability map of the real scene image tampering mask, Up(*) represents the upsampling operation, Sigmoid(*) represents the Sigmoid activation function, SMM(*) represents the feature fusion module, S represents the set of attention feature matrices, S = {s k |k = 1, 2, …, N}; H, W, and E respectively represent the height, width, and number of channels of the attention feature matrix, and E′ represents the unified number of channels of the attention feature matrix;

[0032] A probability threshold is set, and the probability map of the real scene image tampering mask is binarized according to the probability threshold. If the probability of a pixel point in the probability map of the real scene image tampering mask is greater than the probability threshold, the pixel value of the pixel point is set to 1, otherwise it is set to 0, to obtain the real scene image tampering mask map. The real scene image tampering mask map is a binary image, and the area marked as "1" in the binary image, i.e., the white area, is the located tampering area.

[0033] As the depth of the feature extraction module increases, while the semantic information of the initial feature map of the stage extracted becomes abstract, the receptive field is also gradually increasing; the feature attention scales of the full self-attention units in different feature processing branches are also different in the spatial domain; using the outputs of all full self-attention units to decode the real scene image tampering mask map can not only reuse and integrate the multi-stage prediction results, but also simultaneously pay attention to the multi-scale tampering features in the spatial domain, further enriching the relationship information.

[0034] The present invention also provides a real scene image tampering localization method based on the self-attention mechanism. Based on the real scene image tampering localization system based on the self-attention mechanism described in any one of the above, the method includes:

[0035] S1: Obtain a real scene tampered image;

[0036] S2: Perform multi-stage feature extraction on the real scene tampered image to obtain N initial feature maps of the stages;

[0037] S3: Perform downsampling operation, flattening operation, and attention calculation operation on the initial feature map of the Nth stage to obtain the Nth attention feature matrix;

[0038] S4: Perform downsampling operation, flattening operation, addition operation with the attention feature matrix of the subsequent feature processing branch, and attention calculation operation on the initial feature maps of the previous N - 1 stages to obtain the attention feature matrices of the previous N - 1 stages;

[0039] S5: Perform feature fusion on the N attention feature matrices to obtain the real - scene image tampering mask map.

[0040] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0041] The present invention constructs a real - scene image tampering localization system based on the self - attention mechanism. The real - scene tampered image is input into the feature extraction module for multi - stage feature extraction operations to obtain the initial feature maps of N stages. The feature extraction of multiple stages enables the initial feature maps of each stage to have richer semantic information and a larger receptive field, providing rich feature maps for subsequent feature processing. The feature processing module includes N parallel feature processing branches, and each feature processing branch corresponds to processing the initial feature maps of N stages, forming differences in the attention scale. The Nth feature processing branch performs downsampling operation, flattening operation, and attention calculation operation on the initial feature map of the Nth stage to obtain the Nth attention feature matrix. For the previous N - 1 feature processing branches, when performing the attention calculation operation, the output of the full self - attention unit of the subsequent feature processing branch is connected to the input of the full self - attention unit of the previous feature processing branch in the way of matrix addition, comprehensively paying attention to the texture features and tampering traces of the initial feature maps of each stage. The feature fusion module performs feature fusion operations on the N attention feature matrices to achieve reuse and integration, and also pays attention to the multi - scale tampering features in the spatial domain, finally accurately locating the tampering position of the real - scene image. The present invention can achieve accurate tampering localization in real - scene tampered images. Description of the Drawings

[0042] Figure 1 They are example diagrams of two types of scene - tampered images and corresponding tampering masks described in the background art.

[0043] Figure 2 It is a schematic structural diagram of a real - scene image tampering localization system based on the self - attention mechanism described in Embodiment 1.

[0044] Figure 3 It is a schematic structural diagram of a real - scene image tampering localization system based on the self - attention mechanism described in Embodiment 2.

[0045] Figure 4 It is a schematic structural diagram of the full self - attention unit described in Embodiment 2.

[0046] Figure 5 Schematic structural diagram of the hybrid multi-layer perceptron layer described in Embodiment 2.

[0047] Figure 6 Schematic structural diagram of the feature fusion module described in Embodiment 2.

[0048] Figure 7 Schematic comparison diagram of the predicted mask graphs of a real-scene image forgery localization system based on self-attention mechanism and three traditional methods in different real-scene forged images described in Embodiment 2.

[0049] Figure 8 Flowchart of a real-scene image forgery localization method based on self-attention mechanism described in Embodiment 3. Detailed implementation manners

[0050] The accompanying drawings are only for illustrative purposes and should not be construed as limitations on this patent;

[0051] To better illustrate this embodiment, some components in the accompanying drawings are omitted, enlarged or reduced, which do not represent the dimensions of the actual product;

[0052] For those skilled in the art, it is understandable that some well-known structures and their descriptions in the accompanying drawings may be omitted.

[0053] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] Embodiment 1

[0055] The present invention provides a real-scene image forgery localization system based on self-attention mechanism, as shown in Figure 2 , including a feature extraction module, a feature processing module, and a feature fusion module; the feature processing module includes N parallel feature processing branches;

[0056] The real-scene forged image is input into the feature extraction module for multi-stage feature extraction operations to obtain N-stage initial feature maps, and then input into the feature processing module; the Nth feature processing branch performs downsampling operations, flattening operations, and attention calculation operations on the Nth-stage initial feature map to obtain the Nth attention feature matrix, which is input into the (N - 1)th feature processing branch and the feature fusion module; for the first N - 1 feature processing branches, each feature processing branch performs downsampling operations, flattening operations, addition operations with the attention feature matrix of the subsequent feature processing branch, and attention calculation operations on the corresponding stage initial feature map to obtain the attention feature matrix of this feature processing branch, which is input into the previous feature processing branch and the feature fusion module; the feature fusion module performs feature fusion operations on the N attention feature matrices to obtain the real-scene image forgery mask graph.

[0057] In the specific implementation process, this embodiment constructs a real-scene image tampering localization system based on the self-attention mechanism. The real-scene tampered image is input into the feature extraction module for multi-stage feature extraction operations to obtain N-stage initial feature maps. The feature extraction in multiple stages enables the stage initial feature maps to have richer semantic information and a larger receptive field, providing rich feature maps for subsequent feature processing. The feature processing module includes N parallel feature processing branches, and each feature processing branch corresponds to processing the N-stage initial feature maps, forming differences in the attention scale. The Nth feature processing branch performs downsampling operations, flattening operations, and attention calculation operations on the Nth-stage initial feature map to obtain the Nth attention feature matrix. For the first N-1 feature processing branches, when performing the attention calculation operation, the output of the full self-attention unit of the subsequent feature processing branch is connected to the input of the full self-attention unit of the previous feature processing branch in the way of matrix addition, comprehensively paying attention to the texture features and tampering traces of the stage initial feature maps. The feature fusion module performs feature fusion operations on the N attention feature matrices to achieve reuse and integration, and also pays attention to the multi-scale tampering features in the spatial domain, finally accurately locating the tampering position of the real-scene image. This embodiment can achieve accurate tampering localization in real-scene tampered images.

[0058] Embodiment 2

[0059] This embodiment provides a real-scene image tampering localization system based on the self-attention mechanism. As shown in Figure 3, it includes a feature extraction module, a feature processing module, and a feature fusion module. The feature processing module includes N parallel feature processing branches. In this embodiment, N = 4.

[0060] The real-scene tampered image is input into the feature extraction module for multi-stage feature extraction operations to obtain N-stage initial feature maps, and then input into the feature processing module. The Nth feature processing branch performs downsampling operations, flattening operations, and attention calculation operations on the Nth-stage initial feature map to obtain the Nth attention feature matrix, and then inputs it into the (N - 1)th feature processing branch and the feature fusion module. For the first N - 1 feature processing branches, each feature processing branch performs downsampling operations, flattening operations, addition operations with the attention feature matrix of the subsequent feature processing branch, and attention calculation operations on the corresponding stage initial feature map to obtain the attention feature matrix of this feature processing branch, and then inputs it into the previous feature processing branch and the feature fusion module. The feature fusion module performs feature fusion operations on the N attention feature matrices to obtain the real-scene image tampering mask map.

[0061] The feature extraction module includes N sequentially connected convolutional neural network units.

[0062] Real-scene tampered image input feature extraction module. The first convolutional neural network unit extracts features from the real-scene tampered image to obtain the initial feature map f1 of the first stage, and inputs it to the input feature processing module and the second convolutional neural network unit; the second convolutional neural network unit extracts features from the initial feature map f1 of the first stage to obtain the initial feature map f2 of the second stage, and inputs it to the input feature processing module and the third convolutional neural network unit; until the Nth convolutional neural network unit extracts features from the initial feature map f of the N-1th stage N-1 to obtain the initial feature map f of the Nth stage N , and inputs it to the input feature processing module.

[0063] The convolutional neural network units are connected in sequence, increasing the depth of the feature extraction module. While the semantic information of the extracted initial feature maps of each stage becomes more abstract, the receptive field gradually increases.

[0064] The feature processing module includes N parallel feature processing branches; the Nth feature processing branch includes an overlapping convolutional unit, a flattening unit, and a full self-attention unit connected in sequence; the first N-1 feature processing branches have the same structure, each including an overlapping convolutional unit, a flattening unit, a first matrix addition point, and a full self-attention unit connected in sequence; the output end of the full self-attention unit in the jth feature processing branch is also connected to the matrix addition point in the j-1th feature processing branch, j = 2, …, N;

[0065] The overlapping convolutional unit in each feature processing branch is correspondingly connected to the N convolutional neural network units in the feature extraction module, and each full self-attention unit is connected to the feature fusion module;

[0066] The convolutional kernels of the overlapping convolutional units are all 3×3;

[0067] The corresponding initial feature map f k is downsampled by a factor of two through an overlapping convolutional layer with a convolutional kernel size of 3×3 and a stride of 2, and the number of channels is unified to the same dimension E to obtain a downsampled feature map; the flattening unit is used to flatten the corresponding downsampled feature map, restructuring the three-dimensional downsampled feature map into a two-dimensional matrix; the full self-attention unit of the Nth feature processing branch performs a self-attention calculation operation on the corresponding two-dimensional matrix to obtain the Nth attention feature matrix, which is upsampled and then input to the matrix addition point of the N-1th feature processing branch, i.e., the inverse attention connection; in the first N-1 feature processing branches, the matrix addition point is used to add the two-dimensional matrix of this feature processing branch and the attention feature matrix of the subsequent feature processing branch, and then input it to the full self-attention unit of this feature processing branch for a self-attention calculation operation to obtain the attention feature matrix of this feature processing branch;

[0068] The specific method for the fully self-attention unit to calculate the attention feature matrix is as follows:

[0069]

[0070] In the formula, s k represents the k-th attention feature matrix, FBM k represents the k-th fully self-attention unit, represents the k-th overlapping convolution unit, flatten k represents the k-th flattening unit, Up(*) represents the upsampling operation, f k represents the initial feature map of the k-th stage, k ∈ {1, 2, …, N};

[0071] As Figure 4 shown, the fully self-attention unit includes P fully self-attention networks connected in sequence; each fully self-attention network includes a first normalization layer, a multi-head self-attention layer, a second matrix addition point, a second normalization layer, a hybrid multi-layer perceptron layer, and a third matrix addition point connected in sequence; the input end of the first normalization layer is also connected to the input end of the second matrix addition point, and the output end of the second matrix addition point is also connected to the input end of the third matrix addition point.

[0072] As Figure 5 shown, the hybrid multi-layer perceptron layer includes a first fully connected layer, a depthwise separable convolution layer, a second fully connected layer, a global channel pooling layer, a third fully connected layer, a fourth fully connected layer, and a channel multiplication point connected in sequence; the output end of the second fully connected layer is also connected to the input end of the channel multiplication point.

[0073] The hybrid multi-layer perceptron layer further enriches the relationship information on the basis of the conventional multi-layer perceptron. The first fully connected layer, the depthwise separable convolution layer, and the second fully connected layer constitute a convolution-multi-layer perceptron. Introducing a depthwise separable convolution in the multi-layer perceptron to replace the fixed position encoding in the attention mechanism can not only input images of any resolution, improving the applicability of the method, but also introduce local spatial domain information to a certain extent; the global channel pooling layer, the third fully connected layer, the fourth fully connected layer, and the channel multiplication point constitute a channel attention mechanism, which is used to focus on the possible tampered semantic information in the channels from a global scale. First, global channel information is obtained through the global channel pooling layer and activation by the ReLU function. Then, the third fully connected layer and the fourth fully connected layer are connected and activated by the Sigmoid function to calculate the weight vector of each channel at the global scale. Finally, the weight vector and the input of the global channel pooling layer are multiplied and weighted by channels according to the importance of the channels at the global scale.

[0074] The full self-attention unit can focus on tampering features from the global spatial scale, local spatial scale, global channel scale, and local channel scale. The full self-attention unit is composed of P sequentially connected full self-attention networks. Each full self-attention network includes a first normalization layer, a multi-head self-attention layer, a second matrix addition point, a second normalization layer, a hybrid multi-layer perceptron layer, and a third matrix addition point. The multi-head self-attention layer integrates multiple self-attention calculations. The calculation formula is as follows:

[0075]

[0076] MHSA(Z)=[SA1(Z);…;SA m (Z)]W H

[0077]

[0078] Where Z represents the output of the first normalization layer, and the number of channels is E; the self-attention calculation encodes Z into Q, K, V matrices, and calculates the relationship between self-attention features based on the self-learning encoding method; SA(*) represents self-attention calculation, MHSA(*) represents multi-head self-attention splicing, and W H Represents the parameter matrix, m represents the number of relationship heads, and d represents the number of dimensions.

[0079] The full self-attention unit can focus on tampering features from the global spatial scale, local spatial scale, global channel scale, and local channel scale; for the full self-attention units in different feature processing branches, since the initial feature maps of the input stages of different feature processing branches are at different stages, the image semantic information they focus on is different. The full self-attention units in the front feature processing branches tend to focus on texture features in tampered images of real scenes, and the full self-attention units in the back feature processing branches tend to focus on tampering traces; except for the last feature processing branch, the output of the full self-attention unit of the latter feature processing branch is connected to the input of the full self-attention unit of the previous feature processing branch through matrix addition, that is, an inverse attention connection is performed.

[0080] like Figure 6 As shown, the feature fusion module includes a dimension reorganization layer, a fifth fully connected layer, an upsampling layer, a channel splicing layer, a sixth fully connected layer and a seventh fully connected layer connected in sequence.

[0081] The feature fusion module performs feature fusion operation on N attention feature matrices to obtain a tampering mask of a real scene image as follows:

[0082] Input N attention feature matrices into the feature fusion module, the attention feature matrix s k The dimension is The attention feature matrix is reshaped from two - dimensional to three - dimensional by the dimension reshaping layer, and the dimension change is The attention feature matrix is unified to the same number of channels by the fifth fully - connected layer, that is The size of the attention feature matrix is adjusted to the size of the attention feature matrix with the largest size among them, that is, H×W×E′, by the up - sampling layer; the attention feature matrices are concatenated in channels, that is, H×W×NE′, by the channel concatenation layer; the number of channels of the attention feature matrix is shrunk to 1, that is, H×W×1, by the sixth fully - connected layer and the seventh fully - connected layer; finally, after activating the attention feature matrix, the up - sampling operation decodes the probability map of the real - scene image tampering mask:

[0083] M = Up(Sigmoid(SMM(S)))

[0084] In the formula, M represents the probability map of the real - scene image tampering mask, Up(*) represents the up - sampling operation, Sigmoid(*) represents the Sigmoid activation function, SMM(*) represents the feature fusion module, S represents the set of attention feature matrices, S={s k |k = 1, 2, …, N}; H, W, and E respectively represent the height, width, and number of channels of the attention feature matrix, and E′ represents the number of channels after the attention feature matrix is unified.

[0085] In this embodiment, a probability threshold of 0.5 is set, and the probability map of the real - scene image tampering mask is binarized according to the probability threshold. If the probability of a pixel point in the probability map of the real - scene image tampering mask is greater than 0.5, the pixel value of the pixel point is set to 1, otherwise it is set to 0, to obtain the real - scene image tampering mask map. The real - scene image tampering mask map is a binary image, and the area marked as "1" in the binary image, that is, the white area, is the located tampering area.

[0086] As the depth of the feature extraction module increases, while the semantic information of the initial feature map of the stage being extracted becomes more abstract, the receptive field is also gradually increasing; the feature attention scales of the full self - attention units in different feature processing branches in the spatial domain are also different; using the outputs of all full self - attention units to decode the real - scene image tampering mask map can not only reuse and integrate the multi - stage prediction results, but also simultaneously pay attention to the multi - scale tampering features in the spatial domain, further enriching the relationship information.

[0087] In the specific implementation process, in order to balance the tampering localization effect and the consumption of computing resources, so that the localization system has a better tampering localization effect with a lower computational cost, the convolutional neural network unit adopted in this embodiment is ConvNeXt-Tiny, with a quantity of 4, and the encoding dimension E is 256; the feature processing module includes 4 parallel feature processing branches, and the full self-attention unit includes 4 sequentially connected full self-attention networks; the number of multi-head relationships d of the multi-head self-attention layer in the full self-attention network is 4. In the fully connected layer of the hybrid multi-layer perceptron layer, the number of channels will be expanded and then restored to the original number of channels, and the expansion multiple is 4; in the feature fusion module, the upsampling layer adjusts the size of the attention feature matrix to the size of the largest attention feature matrix among them, and E' is 512; during training optimization, the real-scene tampered image dataset of the Alibaba Tianchi Big Data Competition platform is used for training and testing. There are a total of 5405 real-scene tampered images for network training and verification, and 600 are used for testing; it is implemented through programming in the Python language. The neural network framework used is PyTorch, and the operating system is Ubuntu20.04. Training is carried out on a server configured with an NVIDIA RTX 3090 GPU. The optimizer used for training is AdamW. During training and testing, the images are scaled to 512 × 512, but the relevant performance metrics will be calculated at the original image resolution. The training input batch size is 6, and the learning rate is 8×10 -5 , and the learning rate will be multiplied by 0.8 after 12 rounds of iteration, and a total of 200 rounds of training will be carried out. During training, simple data augmentation methods such as cropping, scaling after random cropping, rotation, and flipping are adopted; the input is multiple RGB three-channel real-scene tampered images scaled to 512x512x3, and the final output is a pixel-level tampering probability mask of the tampered area of 512x512x1. The tampering probability mask is used to calculate the loss with the real mask, and the network parameters are updated through the backpropagation algorithm;

[0088] The positioning system of this embodiment is used for comparison with traditional methods; the traditional methods include MVSS-Net, which simultaneously focuses on edge information and noise information through a two-stream residual network, and is the current optimal mainstream tampering localization method; ManTra-Net, which extracts features in the noise domain and the spatial domain and then performs abnormal area detection; RRU-Net, a tampering localization method based on a computer vision segmentation network, which is a circular residual U-shaped segmentation network that strengthens tampering features through feature reuse; the performance evaluation metrics include the harmonic mean F1 of the precision rate and the recall rate, the intersection over union IoU of the predicted area and the tampered area, the MCC coefficient, and the area under the ROC curve AUC. The performance comparison results are shown in the following table:

[0089]

[0090] As can be seen from the above table, the positioning system of this embodiment has achieved the best performance among the four indicators compared with the other three methods. The F1 index is 8.98% higher than that of MVSS-Net, and the IoU index is 9.38% higher than that of MVSS-Net, indicating that the positioning system of this embodiment has a more accurate positioning mask. The AUC index is 3.58% higher than that of MVSS-Net, indicating that the positioning system of this embodiment can make more accurate pixel-level probability predictions.

[0091] As Figure 7 shown, it is a comparison schematic diagram of the predicted mask graphs of the positioning system of this embodiment and three traditional methods on tampered images in different real scenarios; the first row is the comparison schematic diagram of the predicted mask graphs of book cover images, the second and third rows are the comparison schematic diagrams of the predicted mask graphs of certificate images, the fourth row is the comparison schematic diagram of the predicted mask graphs of screenshot images, and the fifth row is the comparison schematic diagram of the predicted mask graphs of document images; the first column is the test image, the second column is the real tampering mask graph, the third column is the predicted mask graph of the positioning system of this embodiment, the fourth column is the predicted mask graph of MVSS-Net, the fifth column is the predicted mask graph of RRU-Net, and the fourth column is the predicted mask graph of ManTra-Net; it can be seen from the figure that the predicted mask graph of the positioning system of this embodiment has a higher similarity with the real tampering mask graph. From the first row, it can be seen that the positioning system of this embodiment can even identify the tampered text content; from the fourth row, it can be seen that the tampering type is simple splicing, and only the positioning system of this embodiment makes an accurate prediction; from the fifth row, it can be seen that the tampering area is small and the positioning difficulty is large, and the positioning system of this embodiment makes an accurate prediction.

[0092] Embodiment 3

[0093] This embodiment provides a real-scene image tampering localization method based on the self-attention mechanism, based on the real-scene image tampering localization system based on the self-attention mechanism described in Embodiment 1 or 2, as Figure 8 shown, the method includes:

[0094] S1: Obtain a real-scene tampered image;

[0095] S2: Perform multi-stage feature extraction on the real-scene tampered image to obtain N-stage initial feature maps;

[0096] S3: Perform downsampling operation, flattening operation, and attention calculation operation on the Nth-stage initial feature map to obtain the Nth attention feature matrix;

[0097] S4: Perform downsampling operation, flattening operation, addition operation with the attention feature matrix of the subsequent feature processing branch, and attention calculation operation on the first N-1 stage initial feature maps to obtain the first N-1 attention feature matrices;

[0098] S5: Feature fusion is performed on N attention feature matrices to obtain a tampering mask map of the real scene image.

[0099] The same or similar reference numerals correspond to the same or similar components;

[0100] The terms used to describe the positional relationship in the drawings are for illustrative purposes only and should not be construed as a limitation of this patent;

[0101] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A real-scene image forgery localization system based on the self-attention mechanism, characterized in that, It includes a feature extraction module, a feature processing module, and a feature fusion module; The real-scene tampered image is input into the feature extraction module for multi-stage feature extraction operations to obtain N-stage initial feature maps, which are then input into the feature processing module; The Nth feature processing branch performs downsampling operations, flattening operations, and attention calculation operations on the Nth-stage initial feature map to obtain the Nth attention feature matrix, which is then input into the (N - 1)th feature processing branch and the feature fusion module; for the first (N - 1) feature processing branches, each feature processing branch performs downsampling operations, flattening operations, addition operations with the attention feature matrix of the subsequent feature processing branch, and attention calculation operations on the corresponding stage initial feature map to obtain the attention feature matrix of this feature processing branch, which is then input into the previous feature processing branch and the feature fusion module; The feature fusion module performs feature fusion operations on the N attention feature matrices to obtain a real-scene image tampering mask map; The feature extraction module includes N sequentially connected convolutional neural network units; The feature processing module includes N parallel feature processing branches; the Nth feature processing branch includes an overlapping convolutional unit, a flattening unit, and a full self-attention unit connected in sequence; the first (N - 1) feature processing branches have the same structure and each includes an overlapping convolutional unit, a flattening unit, a first matrix addition point, and a full self-attention unit connected in sequence; the output end of the full self-attention unit in the jth feature processing branch is also connected to the matrix addition point in the (j - 1)th feature processing branch, where j = 2, …, N; The overlapping convolutional unit in each feature processing branch is correspondingly connected to the N convolutional neural network units in the feature extraction module, and each full self-attention unit is connected to the feature fusion module; The specific method for the full self-attention unit to calculate the attention feature matrix is: where s k represents the k-th attention feature matrix, FAM k represents the k-th full self-attention unit, represents the k-th overlapping convolution unit, flatten k represents the k-th flattening unit, Up(*) represents the upsampling operation, f k represents the initial feature map of the k-th stage, k ∈ {1, 2, …, N}.

2. The real-scene image forgery localization system based on the self-attention mechanism according to claim 1, characterized in that, Real scene tampered image input feature extraction module. The first convolutional neural network unit extracts features from the real scene tampered image to obtain the initial feature map f1 of the first stage, and inputs it to the feature processing module and the second convolutional neural network unit. The second convolutional neural network unit extracts features from the initial feature map f1 of the first stage to obtain the initial feature map f2 of the second stage, and inputs it to the feature processing module and the third convolutional neural network unit. This process continues until the Nth convolutional neural network unit extracts features from the initial feature map f N-1 of the (N - 1)th stage to obtain the initial feature map f N of the Nth stage, and inputs it to the feature processing module.

3. The real-scene image forgery localization system based on the self-attention mechanism according to claim 1, wherein The convolution kernels of the overlapping convolutional units all have a size of 3×3.

4. The real-scene image forgery localization system based on the self-attention mechanism according to claim 1, characterized in that, The full self-attention unit includes P sequentially connected full self-attention networks; each full self-attention network includes a first normalization layer, a multi-head self-attention layer, a second matrix addition point, a second normalization layer, a hybrid multi-layer perceptron layer, and a third matrix addition point connected in sequence; the input end of the first normalization layer is also connected to the input end of the second matrix addition point, and the output end of the second matrix addition point is also connected to the input end of the third matrix addition point.

5. The real-scene image forgery localization system based on the self-attention mechanism according to claim 4, characterized in that, The hybrid multi-layer perceptron layer includes a first fully connected layer, a depthwise separable convolutional layer, a second fully connected layer, a global channel pooling layer, a third fully connected layer, a fourth fully connected layer, and a channel multiplication point connected in sequence; the output end of the second fully connected layer is also connected to the input end of the channel multiplication point.

6. The real-scene image forgery localization system based on the self-attention mechanism according to claim 1, characterized in that, The feature fusion module includes a dimension rearrangement layer, a fifth fully connected layer, an upsampling layer, a channel concatenation layer, a sixth fully connected layer, and a seventh fully connected layer connected in sequence.

7. The real-scene image forgery localization system based on the self-attention mechanism according to claim 6, wherein The specific method for the feature fusion module to perform feature fusion operations on the N attention feature matrices to obtain a real-scene image tampering mask map is: Input N attention feature matrices into the feature fusion module, where the attention feature matrix s k has a dimension of Reshape the attention feature matrix from two-dimensional to three-dimensional through the dimension reshaping layer, with the dimension change being Unify the attention feature matrices to the same number of channels through the fifth fully connected layer, that is Adjust the size of the attention feature matrix to the size of the attention feature matrix with the largest size through the upsampling layer, that is, H×W×E'; splice the channels of the attention feature matrix through the channel splicing layer, that is, H×W×NE'; contract the number of channels of the attention feature matrix to 1 through the sixth fully connected layer and the seventh fully connected layer, that is, H×W×1; finally, activate the attention feature matrix and perform upsampling operations to decode the probability map of the real scene image tampering mask: M = Up(Sigmoid(SMM(S))) where M represents the probability map of real-scene image tampering mask, Up(*) represents the upsampling operation, Sigmoid(*) represents the Sigmoid activation function, SMM(*) represents the feature fusion module, S represents the set of attention feature matrices, S = {s k | k = 1, 2, …, N}; H, W, and E respectively represent the height, width, and number of channels of the attention feature matrix, and E' represents the unified number of channels of the attention feature matrix; Set a probability threshold, and perform a binarization operation on the real-scene image tampering mask probability map according to the probability threshold to obtain a real-scene image tampering mask map.

8. A method for real - scene image forgery localization based on self - attention mechanism, based on the real - scene image forgery localization system based on self - attention mechanism according to any one of claims 1 - 7, characterized in that, The method includes: S1: Obtain a real-scene tampered image; S2: Perform multi-stage feature extraction on the real-scene tampered image to obtain N-stage initial feature maps; S3: Perform downsampling operation, flattening operation, and attention calculation operation on the N-stage initial feature map to obtain the Nth attention feature matrix; S4: Perform downsampling operation, flattening operation, addition operation with the attention feature matrix of the subsequent feature processing branch, and attention calculation operation on the first N-1 stage initial feature maps to obtain the first N-1 attention feature matrices; S5: Perform feature fusion on the N attention feature matrices to obtain a real-scene image tampering mask map.

Images