Multi-tampering localization method based on foreground-background joint modulation and edge guidance

Abstract

The present invention is a multi-tampering positioning method based on foreground-background joint modulation and edge guidance, and the positioning method includes the following contents: obtaining a multi-operation image data set, each image contains at least two types of tampering, and the tampering types include splicing, copy-move and deletion; constructing a multi-tampering positioning model, the multi-tampering positioning model includes a backbone network composed of a Res2net network and a residual network connected to each scale output of the Res2net network, and an edge-assisted refinement fusion module; using the multi-operation image data set to train the multi-tampering positioning model, and the trained multi-tampering positioning model is used for positioning different types of tampering areas in the image. The problem of how to accurately locate multiple tampering areas in an image is solved, and accurate positioning of multiple tampering types of a picture is achieved.

Description

Technical Field

[0001] The technical solution of the present invention relates to the field of deep learning image processing and pattern recognition, and specifically to a multi-tampering positioning method based on foreground-background joint modulation and edge guidance. Background Art

[0002] The purpose of image tampering localization is to detect the part of an image that has been artificially modified. With the popularity of social media and the Internet and the development of deep learning, the development of image editing technology has made tampered images realistic and widely desirable. For example, the latest popular diffusion model, the public can easily generate manipulated images or videos at low cost. Image tampering has become a common means. Tampered images are used to spread false information, mislead the audience, and undermine the reputation of individuals or organizations. This trend poses a serious threat to social stability.

[0003] Image content tampering can generally be divided into three types: splicing, which segments an area of ​​one image and pastes it into another image; copy-paste, which copies and pastes an object from one area of ​​a given image to another area of ​​the image; and removal, which removes an area from an image and repairs the removed area. A large number of tampered images have been generated through these three tampering techniques, but locating multiple tampered areas in a picture remains a challenge.

[0004] In the early days, researchers only detected a certain type of tampering, such as splicing and copy-move. In recent years, general frameworks for multiple types of tampering have also been proposed, such as ManTra-Net, SPAN, MVSS-Net, ObjectFormer, PSCC-Net, and TA-Net. Recently, some algorithms have also been proposed, such as the noise-sensitive tampering localization network TruFor and the CFL-Net with contrast loss, which can better separate tampered areas from untampered areas.

[0005] However, none of the above methods explore the situation where an image has multiple types of tampering. Studies have shown that an image will leave different tampering artifacts when it is spliced, copied-moved, and removed. Common positioning methods do not consider the spatial differences of images with different tampering methods and do not fully utilize frequency domain information. For example, ObjectFormer only uses discrete cosine transform (Discrete Cosine Transform) to extract high-frequency information of the image, and does not consider information in other frequency bands. Finally, the proposed edge method only uses sobel convolution to extract edges, and does not introduce other lightweight methods to improve the efficiency of edge extraction.

[0006] Therefore, a method that can simultaneously handle the localization of multiple tampering types needs to be proposed. Summary of the invention

[0007] In order to solve the problem of multiple tampering areas in an image, the present invention proposes a foreground-background joint adjustment and edge-guided multi-tampering positioning method.

[0008] The technical solution adopted by the present invention to solve the technical problem is:

[0009] In a first aspect, the present invention provides a multi-tampering positioning method based on foreground-background joint adjustment and edge guidance, and the positioning method includes the following contents:

[0010] A multi-operation image dataset is obtained, each image contains at least two tampering types, and the tampering types include splicing, copy-move, and deletion;

[0011] Constructing a multi-tampering localization model, the multi-tampering localization model includes a backbone network consisting of a Res2net network and a residual network connected to each scale output of the Res2net network, and an edge-assisted refinement fusion module;

[0012] The output of each scale of the backbone network is connected to a foreground-background feature separation module, and a mask is obtained by splicing the outputs of the two middle scales of the backbone network and performing a mask operation;

[0013] The foreground-background feature separation module is used to separate foreground information and background information. In the foreground-background feature separation module, the mask is multiplied by the output of the corresponding scale to obtain the foreground information, and the mask is inverted and then multiplied by the output of the corresponding scale to obtain the background information.

[0014] The foreground information is processed by the foreground processing module to obtain the foreground feature f i , the background information is processed by the background processing module to obtain the background feature b i , foreground feature f i and background features b i After splicing, we get the feature F of the foreground and background splicing. i , i=1, 2, 3, 4;

[0015] The edge-assisted refinement and fusion module includes an edge-assisted branch and a refinement and fusion module. The edge-assisted branch processes the features F1 and F4 after the foreground and background are spliced ​​to obtain the predicted edge P edge ; Then F1, F2 and F3 are respectively compared with the predicted edge P edgeMultiply them to get the modified features E1, E2, and E3; input the modified features E1, E2, and E3 and the corresponding inverted modified features 1-E1, 1-E2, and 1-E3 into three refinement and fusion modules, and the output results of the three refinement and fusion modules are recorded as P1, P2, and P3 respectively. At the same time, P3 and P2 are used as the inputs of the second refinement and fusion module and the first refinement and fusion module respectively. The third refinement and fusion module also takes F4 as input, and takes the output P1 of the first refinement and fusion module after 1×1 convolution and sigmoid function processing as the prediction result of the final model;

[0016] A multi-operation image dataset is used to train a multi-tampering localization model, and the trained multi-tampering localization model is used to locate different types of tampered areas in the image.

[0017] Furthermore, the foreground processing module includes a wavelet-like decomposition module, a high-frequency fusion module, a frequency domain correction module and a correlation calculation module;

[0018] The wavelet-like decomposition module is used to decompose the foreground information or background information into high-frequency information F HH , high and low frequency information F HL , low and high frequency information F LH and low frequency information F LL ;

[0019] The high-frequency fusion module is used to fuse high-frequency and high-frequency information, high-low-frequency information, and low-frequency and high-frequency information to obtain high-frequency features;

[0020] The frequency domain correction module is used to utilize the low-frequency information F LL Correct the high-frequency features obtained by the high-frequency fusion module to obtain the corrected features F r ;

[0021] The correlation calculation module includes an autocorrelation calculation using the Pierre coefficient and a percentage pooling layer for calculating the corrected feature F. r Perform correlation calculation to obtain the processed foreground feature f i ;

[0022] The background processing module also includes a wavelet-like decomposition, a high-frequency fusion module and a frequency domain correction module, and the output of the frequency domain correction module is used as the output of the background processing module.

[0023] Furthermore, the edge auxiliary branch includes a sobel operator, spatial attention and channel attention.

[0024] Furthermore, the mask operation includes two serially connected convolution blocks, each convolution block consists of 3×3 convolution, BN and ReLUs activation function; the number of input channels of the first convolution block is the sum of the number of channels of X2 and X3 spatial features, and the number of output channels after processing by the two convolution blocks is 1; after processing by the two convolution blocks, the mask is obtained by normalization operation, ReLUs activation function and Sigmoid activation processing in turn.

[0025] In a second aspect, the present invention provides a multi-tampering positioning method based on foreground-background joint modulation and edge guidance, and the specific steps of the method are:

[0026] The first step is to obtain and preprocess the tampered image data

[0027] Step 1.1, obtain the tampered image dataset:

[0028] A multi-operation image dataset is obtained, each image contains at least two tampering types, and the tampering types include splicing, copy-move, and deletion;

[0029] Step 1.2, tampering data preprocessing:

[0030] The tampered images are preprocessed to obtain an image sequence X, where the image resolution after preprocessing is 256×256, and the preprocessing includes random cropping, random horizontal flipping, random scaling and normalization;

[0031] The second step is to obtain foreground features and background features

[0032] Step 2.1: Build the backbone network:

[0033] The image sequence X obtained in step 1.2 above is input into the Res2net network, and four features of different scales are obtained. Then, they are passed through the residual network to obtain spatial features of four dimensions, which are X1∈R 256×64×64 ,X2∈R 512 ×32×32 ,X3∈R 1024×16×16 and X4∈R 2048×16×16 ; The first dimension is the channel dimension, the second and third dimensions are the spatial dimensions, and then the features are respectively enhanced through the residual network to obtain the spatial features;

[0034] Step 2.2, segmentation mask:

[0035] The spatial feature X2∈R obtained in step 2.1 512×32×32 and X3∈R 1024×16×16Spliced, and passed through two consecutive convolution blocks, each convolution block consists of 3×3 convolution, BN and ReLUs activation function; the number of input channels of the first convolution block is the sum of the number of channels of X2 and X3 spatial features, and the number of output channels after processing by the two convolution blocks is 1; after processing by the two convolution blocks, Sigmoid activation is used to obtain the mask. The process of obtaining the mask is shown in formula (2):

[0036] mask=Sigmoid(ReLU(BN(con(concat(X2,X3))))) (2)

[0037] Among them, con represents the convolution block operation, concat represents the concatenation operation, BN represents the normalization operation, and ReLU represents the ReLUs activation function;

[0038] Step 2.3: Use the foreground and background feature separation module to obtain foreground information and background information:

[0039] The foreground and background feature separation module uses the mask mask to multiply the spatial features X1, X2, X3 and X4 to obtain the corresponding foreground information f i '; At the same time, the mask is inverted and then multiplied with the spatial features X1, X2, X3 and X4 to obtain the corresponding background information b i ';

[0040] The third step is to process the foreground and background information.

[0041] Step 3.1: Use the foreground processing module to process foreground information

[0042] Step 3.1.1: Use the wavelet-like decomposition module to obtain the characteristics of different frequency bands:

[0043] The foreground information obtained in 2.3 is decomposed into high-frequency information F using a wavelet-like decomposition module. HH , high and low frequency information F HL , low and high frequency information F LH and low frequency information F LL , the process is shown in formula (5):

[0044] F HH ,F HL ,F LH ,F LL =LWD(f i '),i∈{1,2,3,4} (5)

[0045] Where LWD(·) represents a wavelet-like decomposition operation, f i ' is the foreground information obtained by separating the foreground and background;

[0046] Step 3.1.2: Strengthen high-frequency information:

[0047] The low-frequency information, high-low-frequency information, and high-frequency information are convolved by 1×1 and 3×3 respectively to obtain the feature F' LH , F' HL and F' HH , then flatten the feature size and adjust the size to hw×c; perform matrix multiplication operation on the adjusted low-frequency and high-frequency features, that is, (F' LH ) Τ ×F' HL , and then perform sigmoid operation to obtain the correlation graph F cor , whose dimension is c×c, and its process is expressed as formula (6):

[0048] F cor =Sigmoid((F' LH ) Τ ×F' HL ) (6)

[0049] Finally, the obtained correlation graph is compared with the adjusted high and high frequency features F' HH Perform matrix multiplication, adjust back to the original tensor size, and then concatenate with the three high-frequency information and obtain the feature Fc after 1×1 convolution, and then obtain the fused high-frequency feature F through 1×1 convolution. H , the process is expressed as formula (7)-(8):

[0050] F c =conv(concat{F LH ,F HL ,F HH}) (7)

[0051] F H =conv(concat((Reshape(Reshape(F' HH )×F cor ),F c ))) (8)

[0052] Among them, concat(·) represents the channel concatenation operation, conv(·) represents the 1×1 convolution operation, and Reshape(·) represents the tensor adjustment operation;

[0053] Step 3.1.3: for the high frequency feature F H To make corrections:

[0054] The low frequency information F LL Using low frequency features F L Indicates that the low-frequency feature F LAnd the obtained high frequency feature F H After 1×1 and 3×3 convolution operations respectively and adjusting the feature tensor size to hw×c, we get F l 、F h , calculate the correlation between the two, extract the features with high correlation, select the most relevant feature S, and then compare it with F h Perform matrix multiplication, and then adjust the tensor and add it to the high-frequency feature F H Concatenate to get feature F R , and finally obtain the corrected feature Fr through a residual structure;

[0055] The residual structure includes two branches, one branch uses 1×1 convolution and Sigmoid (·) function to calculate the attention map; the other branch performs continuous 1×1 and 3×3 convolutions; then, the outputs of the two branches are multiplied to obtain the modified feature F r ;

[0056] Step 3.1.4, calculate the correlation:

[0057] The correlation calculation module includes using the Pierre coefficient to perform autocorrelation calculation, a percentage pooling layer, and selecting Top k , get the processed foreground feature f i ; The process is expressed as formula (14):

[0058] f i =Top k (PP(SC(F r ))) (14)

[0059] Among them, PP(·) represents the percentage pooling layer, and SC(·) represents the autocorrelation calculation;

[0060] Step 3.2: Process the background information and stitch the obtained foreground and background stitching features:

[0061] The background processing module also includes a wavelet-like decomposition, a high-frequency fusion module, and a frequency domain correction module. The output of the frequency domain correction module is used as the output of the background processing module to obtain the processed background information b i ;

[0062] Finally, the obtained foreground information and background information are spliced ​​to obtain the foreground and background spliced ​​features F1, F2, F3 and F4;

[0063] The fourth step is to process the features F1 and F4 after the foreground and background are spliced ​​through the edge auxiliary branch to obtain the predicted edge

[0064] Step 4.1: The features F1 and F4 after the foreground and background are concatenated are respectively subjected to the Sobel operator to extract the rough edges F1' and F4';

[0065] In step 4.2, the attention mechanism is used to refine the edge features. The process is expressed as formula (16)-(17):

[0066]

[0067] F sc =conv(concat(F sp ,F ch )) (17)

[0068] Among them, SAM(·) represents the spatial attention mechanism, and CAM(·) represents the channel attention mechanism;

[0069] Step 4.3: The final predicted edge P is obtained by formula (18): edge :

[0070] P edge =conv(concat(F sc ,F1',F4)) (18)

[0071] The fifth step is to use the refinement fusion module to fuse the multi-scale features and obtain the final prediction results. The different colors in the prediction results represent untampered, spliced ​​tampered, copy-paste tampered, and removed tampered:

[0072] F1, F2 and F3 are respectively compared with the predicted edge P edge Multiply them to get the corrected features E1, E2, and E3. The corrected features E1, E2, and E3 are processed by the refinement and fusion module and the output features are recorded as P1, P2, and P3 respectively. The corrected features are negated to get the negated corrected features 1-E1, 1-E2, and 1-E3. The corrected features E1, E2, and E3 and the corresponding negated corrected features 1-E1, 1-E2, and 1-E3 are input into the three refinement and fusion modules.

[0073] The refinement fusion module includes three parallel refinement modules and three fusion modules. The three input features of the refinement fusion module are processed by a refinement module respectively to obtain enhanced features. F1 r and F1 r and After being processed by the first fusion module, and After being processed by the second fusion module, After being processed by the third fusion module, the output feature P of the current refinement fusion module is obtained.i ; The output P1 of the first refinement fusion module after 1×1 convolution and sigmoid function processing is used as the prediction result of the final model;

[0074] The refinement module includes three Conv-BN-ReLUs layers. The input is first enhanced by the first Conv-BN-ReLUs layer to obtain the feature H i Then, in order to highlight the blurred area, H i With 1-H i Element-wise multiplication is performed, and the value of the blurred area will be large by element-wise multiplication, which can well highlight the blurred area. After that, it is processed by the second conv-BN-ReLUs layer and element-wise multiplication is performed with the input, and then it is concatenated with the input, and finally processed by the third conv-BN-ReLUs layer to obtain the enhanced feature F i r ;

[0075] The fusion module includes a concatenation operation, a Conv-BN-ReLUs layer and a sigmoid function. The process of the three fusion modules is expressed as formulas (22)-(23):

[0076]

[0077] Where Conv(·) represents the Conv-BN-ReLUs layer;

[0078] Step 6: Supervise the prediction results

[0079] Step 6.1: Apply binary cross entropy loss L BCE To supervise the predicted mask, it is expressed as formula (24):

[0080]

[0081] in, represents the truth value, and P i x,y Represents the prediction result of the tampered image; l∈{0,1,2,3} represents different tampering categories: real, splicing tampering, copy-paste tampering and removal tampering; Pr represents the probability;

[0082] Step 6.2, apply L dice To detect the predicted edge, it is expressed as formula (25):

[0083]

[0084] in, Represents the prediction result of each pixel in the prediction edge, Represents the true value of each pixel, H and W are the height and width of the image;

[0085] The total loss during training is the marginal prediction loss L dice and the mask prediction loss L BCE sum;

[0086] A multi-operation image dataset is used to train a multi-tampering localization model, and the trained multi-tampering localization model is used to locate different types of tampered areas in the image.

[0087] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which can implement the steps of the method when executed by a processor.

[0088] Compared with the existing method, the present invention has the following beneficial effects:

[0089] The outstanding substantive features of the present invention are: the method of the present invention uses the Res2net network and the residual network as the backbone, and then separates the background and foreground of the feature, and then processes the foreground background information respectively, so as to better focus on the tampering traces in the foreground background; for the foreground information, the foreground feature is decomposed into high-frequency, high-low frequency, low-high frequency and low-low frequency by using the wavelet-like decomposition. Since the tampered area will be hidden in the high-frequency information, the high-frequency fusion module is used for high-frequency, high-low frequency and low-high frequency to further process the high-frequency information. At the same time, the low-frequency information will contain more semantic information, and the low-frequency information is used to assist in correcting the high-frequency features, which is conducive to identifying accurate tampering artifact information. A correlation calculation module is also added to the foreground processing module, which can effectively improve the network's positioning effect on the copy-paste tampering area. The correlation calculation module is subtracted from the background processing module, and the calculation efficiency is improved under the premise of ensuring the positioning effect. At the same time, the features after the foreground and background splicing are processed by the edge-assisted refinement module, and the final accurate prediction results can be obtained.

[0090] The significant improvements of the present invention are:

[0091] (1) The present invention solves the problem of how to accurately locate multiple tampering areas in an image based on a multi-tampering localization method based on foreground-background joint modulation and edge guidance, thereby achieving accurate localization of multiple tampering types in one image.

[0092] (2) Different tampering methods exist in different spaces of the image. For example, splicing and copy-move usually appear in the foreground, while after the removal operation, the tampering will be hidden in the background. The present invention uses this feature to detect and locate images with multiple tampering types, highlighting the differences between different tampering types.

[0093] (3) In the edge-assisted refinement and fusion module of the present invention, the edge-assisted branch is used to improve the positioning accuracy of the network, and the refinement and fusion module is used to efficiently fuse the features.

[0094] (4) The experimental results of the present invention show that MMID is not only helpful for training the identification model, but also has obvious advantages in accuracy and robustness when performing forgery localization and tampering classification simultaneously.

[0095] (5) Previous methods only used high-frequency information and ignored information in other frequency bands. However, low-frequency information contains a large amount of semantic information. Using low-frequency information can enhance and retain the semantic features of the image, thereby improving the accuracy of feature representation. In addition, the semantic information of low-frequency information can help high-frequency features better identify noise information, thereby better discovering the details of tampering artifacts. Low-frequency information contains more semantic information, and low-frequency information can be used to assist high-frequency information in identifying accurate tampering artifact information.

[0096] (6) For the edge auxiliary branch, the present invention extracts edges through a 3×3×2 Sobel operator, and introduces a spatial and channel attention mechanism to improve the effect of edge extraction. The features F1 and F4 after the foreground and background are spliced ​​are respectively subjected to a 3×3×2 Sobel operator to extract rough edges, and then are respectively subjected to a spatial attention mechanism and a channel attention mechanism to obtain spatial attention features and spatial attention features, and the spatial channel attention features are inverted to make the network pay more attention to the edge area and improve the accuracy of edge recognition. The spatial attention features and channel attention features are then used to perform matrix multiplication with the spatial features to obtain refined edge information. Finally, the processed features are spliced ​​together, and the number of channels is adjusted through a 1×1 convolution operation to obtain the final predicted edge.

[0097] (7) For the refinement fusion module, the present invention performs element-wise multiplication of the feature and its inverse feature to emphasize the local details of the difficult-to-tamper area and effectively fuse the features at all levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0099] Figure 1 It is a schematic diagram of the overall process of the multi-tampering positioning method based on foreground-background joint modulation and edge guidance of the present invention.

[0100] Figure 2 It is a structural schematic diagram of the foreground processing module of the present invention.

[0101] Figure 3 It is a structural diagram of the background processing module of the present invention.

[0102] Figure 4It is a structural schematic diagram of the high frequency fusion module of the present invention.

[0103] Figure 5 It is a structural schematic diagram of the frequency domain correction module of the present invention.

[0104] Figure 6 It is a structural schematic diagram of the edge auxiliary branch of the present invention.

[0105] Figure 7 It is a structural schematic diagram of the refinement fusion module of the present invention.

[0106] Figure 8 This is a comparison chart of the prediction results. DETAILED DESCRIPTION

[0107] Figure 1 As shown in the embodiment, the overall process of the multi-tampering positioning method based on foreground-background joint adjustment and edge guidance of the present invention is as follows:

[0108] The preprocessed tampered image is first input into the Res2net network to extract spatial information of different scales, and then it is successively passed through the residual network R to enhance the spatial information to obtain features X1, X2, X3 and X4; secondly, the spatial features of X2 and X3 are used to obtain the mask mask through mask operation, and then the foreground information and background information are obtained through the foreground and background feature separation module, and then the foreground processing module and the background processing module are successively entered to obtain the foreground feature f i and background features b i ; Foreground feature f i and background features b i After splicing, we get the feature F of the foreground and background splicing. i , i = 1, 2, 3, 4, enter the edge-assisted refinement and fusion module;

[0109] The edge-assisted refinement and fusion module includes an edge-assisted branch and a refinement and fusion module. The edge-assisted branch processes the features F1 and F4 after the foreground and background are spliced ​​to obtain the predicted edge P edge ; Then F1, F2 and F3 are respectively compared with the predicted edge P ed ge are multiplied to obtain the modified features E1, E2, and E3; the modified features E1, E2, and E3 and the corresponding inverted modified features 1-E1, 1-E2, and 1-E3 are input into three refinement and fusion modules, and the output results of the three refinement and fusion modules are recorded as P1, P2, and P3, respectively. At the same time, P3 and P2 are used as the inputs of the second refinement and fusion module and the first refinement and fusion module, respectively. The third refinement and fusion module also takes F4 as input, and the output P1 of the first refinement and fusion module is processed by 1×1 convolution and sigmoid as the prediction result of the final model; the edge prediction loss L is introduced. dice and the mask prediction loss LBCE Constrain the model training process.

[0110] Among them, the mask operation includes two serially connected convolution blocks, each convolution block consists of 3×3 convolution, BN and ReLUs activation function; the number of input channels of the first convolution block is the sum of the number of channels of X2 and X3 spatial features, and the number of output channels after the two convolution blocks are processed is 1; after the two convolution blocks are processed, they are subjected to normalization operation, ReLUs activation function and Sigmoid activation processing in turn to obtain the mask.

[0111] Figure 2 As shown in the embodiment, the overall process of the foreground processing module in the present invention is as follows:

[0112] The foreground processing module includes a wavelet-like decomposition module, a high-frequency fusion module, a frequency domain correction module and a correlation calculation module; the wavelet-like decomposition module is used to decompose the foreground information or background information into high-frequency information F HH , high and low frequency information F HL , low and high frequency information F LH and low frequency information F LL ;

[0113] In the high frequency fusion module (see Figure 4 ), the low-frequency information, high-low-frequency information, and high-frequency information are convolved by 1×1 and 3×3 respectively to obtain the feature F L ' H 、F H ' L and F H ' H , and then flatten the feature tensor to hw×c. The flattened feature F' LH , F' HL Perform matrix multiplication, that is, (F' LH ) Τ ×F' HL , and then perform sigmoid operation to obtain the correlation graph F cor Finally, the obtained correlation graph is compared with F' HH The result after the flattening operation is matrix multiplied and then adjusted back to the original tensor size. It is then concatenated with the feature Fc obtained by splicing the three high-frequency information and processing it with 1×1 convolution. The fused high-frequency feature F is then convolved with it with 1×1 convolution. H .

[0114] In the frequency domain correction module (see Figure 5 ), the low-frequency feature F L And the high-frequency feature F obtained by the high-frequency fusion module H After 1×1 and 3×3 convolution operations and flattening, the feature F is obtained. land F h , flattened to hw×c, and then the correlation between the two is calculated to extract the features with high correlation between high-frequency information and low-frequency information, and select the most relevant features. h Perform matrix multiplication and then adjust the tensor size to match the high-frequency feature F H Splice to get F R , and finally obtain the corrected feature F through a residual structure r . For the corrected feature F r Perform correlation calculation to obtain foreground information f i .

[0115] Figure 3 As shown in the embodiment, the background processing module of the present invention lacks the correlation calculation module compared with the foreground information processing module, and the other structures are the same.

[0116] Figure 6 As shown in the example, the overall process of the edge auxiliary branch of the present invention is as follows: the edge auxiliary branch includes a sobel operator, spatial attention and channel attention,

[0117] The present invention uses a 3×3×2 Sobel operator to extract edges, and introduces a spatial and channel attention mechanism to improve the effect of edge extraction. The features F1 and F4 after the foreground and background are spliced, respectively, through a 3×3×2 Sobel operator to extract rough edges to obtain features F1' and F4', and then through a spatial attention mechanism and a channel attention mechanism to obtain spatial attention features and channel attention features f sp and f ch , and the spatial channel attention feature is inverted to obtain 1-f sp and 1-f ch The network pays more attention to the edge area and improves the accuracy of edge recognition. Then the spatial attention features and channel attention features are matrix multiplied with the extracted features to obtain the refined edge information. Finally, the processed features are concatenated step by step, and the number of channels is adjusted through a 1×1 convolution operation to obtain the final predicted edge P. edge .

[0118] Figure 7 As shown in the example, the overall process of the refinement fusion module of the present invention is as follows:

[0119] The refinement fusion module includes three parallel refinement modules and three fusion modules. The three input features of the refinement fusion module are processed by a refinement module respectively to obtain enhanced features. F1 r and F1 r and After being processed by the first fusion module, and After being processed by the second fusion module, After being processed by the third fusion module, the output feature P of the current refinement fusion module is obtained. i ;

[0120] The refinement module includes three Conv-BN-ReLUs layers. The input is first enhanced by the first Conv-BN-ReLUs layer to obtain the feature H i Then, in order to highlight the blurred area, H i With 1-H i Element-wise multiplication is performed, and the value of the blurred area will be large by element-wise multiplication, which can well highlight the blurred area. After that, it is processed by the second conv-BN-ReLUs layer and element-wise multiplication is performed with the input, and then it is concatenated with the input, and finally processed by the third conv-BN-ReLUs layer to obtain the enhanced feature F i r .

[0121] The fusion module consists of a Conv-BN-ReLUs layer and a sigmoid function.

[0122] The present invention will F i With P edge The corrected feature E obtained after multiplication i , negate and modify feature 1-E i And the output feature P obtained by the previous refinement fusion module i+1 Input to the refinement and fusion module, each feature input to the refinement and fusion module passes through the refinement module. The third input of the third refinement and correction module is F4, denoted as P4.

[0123] Example 1

[0124] The present invention provides a multi-tampering positioning method based on foreground-background joint modulation and edge guidance, and the specific steps of the method are:

[0125] The first step is to obtain and preprocess the tampered image data

[0126] Step 1.1, obtain the tampered image dataset:

[0127] The present invention establishes a multi-operation image dataset (MMID) containing 2,000 multi-operation images. Each image contains at least two types of tampering. The tampering types include splicing, copy-move and deletion. The original images are from COCO, and each original image has been manually tampered with by Photoshop software. For the splicing tampering type, the present invention randomly selects two images with similar scenes from the COCO dataset. Using the quick selection tool, select an appropriate object from one of the images, rotate and scale the selected object, and move it to the target image. Finally, perform post-processing operations such as adjusting brightness, contrast, or using a color balance tool to make the tampered area appear more natural. The copy-move tampering type follows similar steps, except that the process is performed on a single image. For the removal tampering type, an object in the image is randomly selected and removed using the stamp tool and the healing brush.

[0128] In order to evaluate the effectiveness of the model, the present invention establishes a single manipulation image dataset (SMID). It includes 1196 spliced ​​images, 208 deleted images and 1465 copy-move images from CASIA, CoMoFoD and NIST16. CASIA and NIST16 contain three types of tampering: splicing, copy-move and deletion, while CoMoFoD only has copy-move.

[0129] The present invention constructs a synthetic data set, which contains 50012 images, each of which has a tampered area. It includes 10000 spliced ​​tampered images automatically generated by a program, wherein the original image is from COCO, 20304 copy-move tampered images from BusterNet, and 19708 deletion tampered images from DEFACTO. Since the MMID data set lacks enough images to train a model with more parameters, pre-training is performed on the synthetic data set, and fine-tuning is performed on the MMID data set after pre-training, and testing is performed using data in a multi-operation image data set and a single-operation image data set. The combination of the two data sets can solve the problem of insufficient labeled data.

[0130] Step 1.2, tampering data preprocessing:

[0131] The tampered images are preprocessed to obtain an image sequence X, where the image resolution after preprocessing is 256×256, and the preprocessing includes random cropping, random horizontal flipping, random scaling and normalization;

[0132] Figure 1 The true value image in is a pixel annotation map of different tampering types after the preprocessing in step 1.2, which indicates the true value of each pixel and is used for subsequent training.

[0133] The second step is to obtain foreground features and background features.

[0134] Step 2.1: Build the backbone network:

[0135] The image sequence X obtained in step 1.2 above is input into the Res2net network, and four features of different scales are obtained. Then, they are passed through the residual network to obtain spatial features of four dimensions, which are X1∈R 256×64×64 ,X2∈R 512 ×32×32 ,X3∈R 1024×16×16 and X4∈R 2048×16×16 The first dimension is the channel dimension, and the second and third dimensions are the spatial dimensions. The features are then respectively enhanced through the residual network to obtain the enhanced spatial features. The process is shown in formula (1):

[0136] {Xi,i=1,2,3,4}←Res(Res2net(X)) (1)

[0137] Where Xi represents the features of the image sequence after being processed by the Res2net network and the residual network in turn;

[0138] Step 2.2, get the segmentation mask:

[0139] The spatial feature X2∈R obtained in step 2.1 512×32×32 and X3∈R 1024×16×16 Spliced, and passed through two consecutive convolution blocks, each of which consists of 3×3 convolution, BN and ReLUs activation function. The number of input channels of the first convolution block is the sum of the number of channels of the X2 and X3 spatial features, and the number of output channels after the two convolution blocks are processed is 1; after the two convolution blocks are processed, the mask is obtained using Sigmoid activation. After the foreground and background feature separation, the mask interacts with the corresponding spatial features. The process of obtaining the mask is shown in formula (2):

[0140] mask=Sigmoid(ReLU(BN(con(concat(X2,X3))))) (2)

[0141] Among them, con represents the convolution block operation, concat represents the concatenation operation, BN represents the normalization operation, and ReLU represents the ReLUs activation function;

[0142] Step 2.3: Use the foreground and background feature separation module to obtain foreground information and background information:

[0143] The foreground and background feature separation module uses the mask obtained in step 2.2 to multiply the spatial features X1, X2, X3 and X4 to obtain the corresponding foreground information f i'; At the same time, the mask is inverted and then multiplied with the spatial features X1, X2, X3 and X4 to obtain the corresponding background information b i '. The process is shown in formula (3)-(4):

[0144] f i '=Xi⊙mask (3)

[0145] b i '=Xi⊙(1-mask) (4)

[0146] The third step is to process the foreground and background information.

[0147] Step 3.1: Use the foreground processing module to process foreground information

[0148] The foreground processing module includes a wavelet-like decomposition module, a high-frequency fusion module, a frequency domain correction module and a correlation calculation module;

[0149] Step 3.1.1: Use the wavelet-like decomposition module to obtain the characteristics of different frequency bands:

[0150] The foreground information obtained in 2.3 is decomposed into high-frequency information F using a wavelet-like decomposition module. HH , high and low frequency information F HL , low and high frequency information F LH and low frequency information F LL The process is shown in formula (5).

[0151] F HH ,F HL ,F LH ,F LL =LWD(f i '),i∈{1,2,3,4} (5)

[0152] Where LWD(·) represents a wavelet-like decomposition operation, f i ' is the foreground information obtained by separating the foreground and background.

[0153] Step 3.1.2: Strengthen high-frequency information:

[0154] Since the tampered area will be hidden in the high-frequency information, the high-frequency fusion module is used to further process the high-frequency information, high-low-frequency information, and low-high-frequency information. The low-high-frequency information, high-low-frequency information, and high-high-frequency information are convolved with 1×1 and 3×3 respectively to obtain the feature F' LH , F' HL and F' HH , and then flatten the feature tensor size and adjust the size to hw×c. Perform matrix multiplication operation on the adjusted low-frequency features and high-frequency features, that is, (F'LH ) Τ ×F' HL , and then perform sigmoid operation to obtain the correlation graph F cor , whose dimension is c×c, and the process can be expressed as formula (6):

[0155] F cor =Sigmoid((F L ' H ) Τ ×F H ' L ) (6)

[0156] Finally, the obtained correlation graph is matrix multiplied with the adjusted high and high frequency features, and then adjusted back to the original tensor size. After that, it is concatenated with the feature Fc obtained by splicing the three high frequency information and processing it by 1×1 convolution, and then convolved by 1×1 convolution to obtain the fused high frequency feature F H The process can be expressed as formula (7)-(8):

[0157] F c =conv(concat{F LH ,F HL ,F HH}) (7)

[0158] F H =conv(concat((Reshape(Reshape(F' HH )×F cor ),F c ))) (8)

[0159] Among them, concat(·) represents the channel concatenation operation, conv(·) represents the 1×1 convolution operation, and Reshape(·) represents the tensor adjustment operation. In formula (8), the inner layer Reshape(·) means changing the size of the feature map from h×w×c to hw×c, and the outer layer Reshape(·) means changing the size of the feature map from hw×c to h×w×c.

[0160] Step 3.1.3: Use the frequency domain correction module to correct the high-frequency features F H To make corrections:

[0161] Previous methods only used high-frequency information and ignored low-frequency information. However, low-frequency information contains a lot of semantic information. Using low-frequency information can enhance and retain the semantic features of the image, thereby improving the accuracy of feature representation. In addition, the semantic information of low-frequency information can help high-frequency features better identify noise information, thereby better discovering the details of tampering artifacts. The frequency domain correction module uses low-frequency information to assist high-frequency features in identifying accurate tampering artifact information.

[0162] Specifically, the low-frequency information F LL Using low frequency features F L Indicates that the low-frequency feature F L And the obtained high frequency feature F H After 1×1 and 3×3 convolution operations respectively and adjusting the feature tensor size to hw×c, we get F l 、F h , calculate the correlation between the two, extract the features with high correlation, select the most relevant feature S, and then compare it with F h Perform matrix multiplication, and then adjust the tensor and add it to the high-frequency feature F H Concatenate to get feature F R , and finally obtain the corrected feature Fr through a residual structure;

[0163] The residual structure includes two branches, one branch uses 1×1 convolution and Sigmoid (·) function to calculate the attention map; the other branch performs continuous 1×1 and 3×3 convolutions; then, the outputs of the two branches are multiplied to obtain the modified feature F r ;

[0164] The process of the frequency domain correction module can be expressed as formula (9-13):

[0165] F h =conv(Conv(F H )),F l =conv(Conv(F L )) (9)

[0166] M=sim(F h ,F l ) (10)

[0167] S=Top1(M) (11)

[0168] F R =concat(F H ,Reshape(F h ×S)) (12)

[0169] F r=concat(Conv(conv(F R ))⊙sigmoid(conv(F R ))) (13)

[0170] Among them, conv(·) represents a 1×1 convolution operation, Conv(·) represents a 3×3 convolution operation, sim(·) represents an operation of calculating correlation using matrix multiplication, Top1(·) represents an operation of selecting the most relevant channel; M represents a feature with high correlation; S represents the most relevant feature.

[0171] Step 3.1.4, perform correlation calculation:

[0172] Finally, in order to improve the network's positioning effect on the copy-paste tampered area, a correlation calculation module is added to the end of the foreground processing module. The correlation calculation module includes autocorrelation calculation using the Pierre coefficient and a percentage pooling layer. The result after the percentage pooling layer is processed by selecting Top k , get the processed foreground feature f i ; The process can be expressed as formula (14):

[0173] f i =Top k (PP(SC(F r ))) (14)

[0174] Among them, PP(·) represents the percentage pooling layer, SC(·) represents the autocorrelation calculation, and the correlation parameter is obtained by calculating the Pierre coefficient. In this embodiment, Top k The value of k is 16 and a 16-channel correlation feature is obtained;

[0175] Step 3.2: Use the background processing module to process the background information and stitch the obtained foreground and background stitching features:

[0176] The background processing module also includes a wavelet-like decomposition, a high-frequency fusion module, and a frequency domain correction module. The output of the frequency domain correction module is used as the output of the background processing module to obtain the processed background information b i , which is similar to processing foreground information.

[0177] Finally, the obtained foreground information and background information are spliced ​​together to obtain the foreground and background spliced ​​features F1, F2, F3 and F4.

[0178] The fourth step is to process the features F1 and F4 after the foreground and background are spliced ​​through the edge auxiliary branch to obtain the predicted edge

[0179] The edge auxiliary branch includes a sobel operator, spatial attention, and channel attention.

[0180] Step 4.1: Use the Sobel operator to get rough edge information:

[0181] The features F1 and F4 after the foreground and background are concatenated are respectively subjected to the Sobel operator to extract the rough edges F1' and F4'. The process can be expressed as formula (15):

[0182] F1'=sobel(F1),F4'=sobel(F4) (15)

[0183] Among them, sobel(·) represents the operation of using the sobel operator to extract edges, sobel-conv3*3*2, the third dimension 2 represents the number of channels of the Sobel operator (Channels), that is, it contains two independent convolution kernels, one for detecting horizontal edges (Gx) and the other for detecting vertical edges (Gy).

[0184] Step 4.2: Use the attention mechanism to refine edge features:

[0185] After that, the spatial attention mechanism and channel attention mechanism are used to obtain the spatial attention feature and spatial attention feature, and the spatial channel feature is inverted to make the network pay more attention to the edge area and improve the accuracy of edge recognition. The process can be expressed as formula (16-17):

[0186]

[0187] F sc =conv(concat(F sp ,F ch )) (17)

[0188] Among them, SAM(·) represents the spatial attention mechanism and CAM(·) represents the channel attention mechanism.

[0189] In step 4.3, the final predicted edge is obtained, and the process can be expressed as formula (18):

[0190] P edge =conv(concat(F sc ,F1',F4)) (18)

[0191] The fifth step is to use the refinement fusion module to fuse the multi-scale features and obtain the final output result P j , j = 1, 2, 3, the result is a tensor of size R H×W×4 The prediction results of , different colors represent untampered (black), spliced ​​tampered (pink), copy-paste tampered (white) and removed tampered (green):

[0192] The present invention combines the features F1, F2 and F3 obtained in step 3.2 with the predicted edge P obtained in step 4. edge Multiply them to get the modified features E1, E2, and E3. The modified features E1, E2, and E3 are processed by the refinement fusion module and the output features are recorded as P1, P2, and P3 respectively. The process of obtaining the modified features can be expressed as formula (19):

[0193] E i =P edge ×F i ,i∈{1,2,3} (19)

[0194] F i With P edge The feature E obtained after multiplication i 、1-E i And the output feature P obtained by the previous refinement fusion module i+1 Input to the refinement and fusion module, each feature input to the refinement and fusion module passes through the refinement module.

[0195] The refinement fusion module includes three parallel refinement modules and three fusion modules. The three input features of the refinement fusion module are processed by a refinement module respectively to obtain enhanced features. F1 r and F1 r and After being processed by the first fusion module, and After being processed by the second fusion module, After being processed by the third fusion module, the output feature P of the current refinement fusion module is obtained. i .

[0196] The refinement module includes three Conv-BN-ReLUs layers. The input is first enhanced by the first Conv-BN-ReLUs layer to obtain the feature H i Then, in order to highlight the blurred area, H i With 1-H i Element-wise multiplication is performed, and the value of the blurred area will be large by element-wise multiplication, which can well highlight the blurred area. After that, it is processed by the second Conv-BN-ReLUs layer and element-wise multiplication is performed with the input, and then it is concatenated with the input, and finally processed by the third Conv-BN-ReLUs layer to obtain the enhanced feature F i r The process can be expressed as formula (20-21):

[0197]

[0198] Here, Conv(·) represents the Conv-BN-ReLUs layer.

[0199] The fusion module includes a concatenation operation, a Conv-BN-ReLUs layer, and a sigmoid function. The output of the third fusion module is P i The process of the three fusion modules can be expressed as formula (22-23):

[0200]

[0201] Here, Conv(·) represents the Conv-BN-ReLU layer.

[0202] The corrected features E1, E2, E3 and the corresponding inverted corrected features 1-E1, 1-E2, 1-E3 are input into three refinement and fusion modules. The output results of the three refinement and fusion modules are recorded as P1, P2, P3, respectively. At the same time, P3 and P2 are used as the input of the second refinement and fusion module and the first refinement and fusion module respectively. The third refinement and fusion module also takes F4 as input, and the output P1 of the first refinement and fusion module after 1×1 convolution and sigmoid function processing is used as the prediction result of the final model.

[0203] Step 6: Supervise the prediction results

[0204] Step 6.1: Apply binary cross entropy loss L BCE To supervise the predicted mask, it can be expressed as formula (24):

[0205]

[0206] in, represents the truth value, and P i x,y represents the prediction result of the tampered image, which is the pixel value in Pi, l∈{0,1,2,3} represents different tampering categories (real, splicing tampering, copy-paste tampering and removal tampering); Pr represents the probability;

[0207] Step 6.2, apply L dice To detect the predicted edge, it can be expressed as formula (25):

[0208]

[0209] in, Represents the prediction result of each pixel in the prediction edge, Represents the true value of each pixel, H and W are the height and width of the image;

[0210] The total loss during training is the marginal prediction loss L dice and the mask prediction loss L BCE The sum of is shown in the following formula (26):

[0211]

[0212] Among them, the value of λ is 3.

[0213] A multi-operation image dataset is used to train a multi-tampering localization model, and the trained multi-tampering localization model is used to locate different types of tampered areas in the image. Figure 8 This is a comparison chart of the positioning results of the method of the present invention for different types of tampering in the same image.

[0214] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A multi-tampering positioning method based on foreground-background joint modulation and edge guidance, characterized in that: The positioning method includes the following contents: A multi-operation image dataset is obtained, each image contains at least two tampering types, and the tampering types include splicing, copy-move, and deletion; Constructing a multi-tampering localization model, the multi-tampering localization model includes a backbone network consisting of a Res2net network and a residual network connected to each scale output of the Res2net network, and an edge-assisted refinement fusion module; The output of each scale of the backbone network is connected to a foreground-background feature separation module, and a mask is obtained by splicing the outputs of the two middle scales of the backbone network and performing a mask operation; The foreground and background feature separation module is used to separate foreground information and background information. In the foreground and background feature separation module, the mask is multiplied by the output of the corresponding scale to obtain the foreground information, and the mask is inverted and then multiplied by the output of the corresponding scale to obtain the background information; The foreground information is processed by the foreground processing module to obtain the foreground feature f i , the background information is processed by the background processing module to obtain the background feature b i , foreground feature f i and background features b i After splicing, we get the feature F of the foreground and background splicing. i , i=1, 2, 3, 4; The edge-assisted refinement and fusion module includes an edge-assisted branch and a refinement and fusion module. The edge-assisted branch processes the features F1 and F4 after the foreground and background are spliced ​​to obtain the predicted edge P edge ; Then F1, F2 and F3 are respectively compared with the predicted edge P edge Multiply them to get the modified features E1, E2, and E3; input the modified features E1, E2, and E3 and the corresponding inverted modified features 1-E1, 1-E2, and 1-E3 into three refinement and fusion modules, and the output results of the three refinement and fusion modules are recorded as P1, P2, and P3 respectively. At the same time, P3 and P2 are used as the inputs of the second refinement and fusion module and the first refinement and fusion module respectively. The third refinement and fusion module also takes F4 as input, and takes the output P1 of the first refinement and fusion module after 1×1 convolution and sigmoid function processing as the prediction result of the final model; A multi-operation image dataset is used to train a multi-tampering localization model, and the trained multi-tampering localization model is used to locate different types of tampered areas in the image.

2. The method according to claim 1, characterized in that The foreground processing module includes a wavelet-like decomposition module, a high-frequency fusion module, a frequency domain correction module and a correlation calculation module; The wavelet-like decomposition module is used to decompose the foreground information or background information into high-frequency information F HH , high and low frequency information F HL , low and high frequency information F LH and low frequency information F LL ; The high-frequency fusion module is used to fuse high-frequency and high-frequency information, high-low-frequency information, and low-frequency and high-frequency information to obtain high-frequency features; The frequency domain correction module is used to utilize the low-frequency information F LL Correct the high-frequency features obtained by the high-frequency fusion module to obtain the corrected features F r ; The correlation calculation module includes an autocorrelation calculation using the Pierre coefficient and a percentage pooling layer for calculating the corrected feature F. r Perform correlation calculation to obtain the processed foreground feature f i ; The background processing module also includes a wavelet-like decomposition, a high-frequency fusion module and a frequency domain correction module, and the output of the frequency domain correction module is used as the output of the background processing module.

3. The method according to claim 1, characterized in that The edge auxiliary branch includes a sobel operator, spatial attention and channel attention.

4. The method according to claim 1, characterized in that The mask operation includes two convolution blocks connected in series, each of which consists of a 3×3 convolution, a BN and a ReLUs activation function; the number of input channels of the first convolution block is the sum of the number of channels of the X2 and X3 spatial features, and the number of output channels after the two convolution blocks are processed is 1; after the two convolution blocks are processed, they are subjected to normalization operation, ReLUs activation function, and Sigmoid activation processing in turn to obtain a mask.

5. A multi-tampering positioning method based on foreground-background joint modulation and edge guidance, characterized in that: The specific steps of the method are: The first step is to obtain and preprocess the tampered image data Step 1.1, obtain the tampered image dataset: A multi-operation image dataset is obtained, each image contains at least two tampering types, and the tampering types include splicing, copy-move, and deletion; Step 1.2, tampering data preprocessing: The tampered images are preprocessed to obtain an image sequence X, where the image resolution after preprocessing is 256×256, and the preprocessing includes random cropping, random horizontal flipping, random scaling and normalization; The second step is to obtain foreground features and background features Step 2.1: Build the backbone network: The image sequence X obtained in step 1.2 above is input into the Res2net network, and four features of different scales are obtained. Then, they are passed through the residual network to obtain spatial features of four dimensions, which are X1∈R 256×64×64 ,X2∈R 512×32×32 ,X3∈R 1024×16×16 and X4∈R 2048×16×16 ; The first dimension is the channel dimension, the second and third dimensions are the spatial dimensions, and then the features are respectively enhanced through the residual network to obtain the spatial features; Step 2.2, segmentation mask: The spatial feature X2∈R obtained in step 2.1 512×32×32 and X3∈R 1024×16×16 Spliced, and passed through two consecutive convolution blocks, each convolution block consists of 3×3 convolution, BN and ReLUs activation function; the number of input channels of the first convolution block is the sum of the number of channels of X2 and X3 spatial features, and the number of output channels after processing by the two convolution blocks is 1; after processing by the two convolution blocks, Sigmoid activation is used to obtain the mask. The process of obtaining the mask is shown in formula (2): mask=Sigmoid(ReLU(BN(con(concat(X2,X3))))) (2) Among them, con represents the convolution block operation, concat represents the concatenation operation, BN represents the normalization operation, and ReLU represents the ReLUs activation function; Step 2.3: Use the foreground and background feature separation module to obtain foreground information and background information: The foreground and background feature separation module uses the mask mask to multiply the spatial features X1, X2, X3 and X4 to obtain the corresponding foreground information f i '; At the same time, the mask is inverted and then multiplied with the spatial features X1, X2, X3 and X4 to obtain the corresponding background information b′ i ; The third step is to process the foreground and background information. Step 3.1: Use the foreground processing module to process foreground information Step 3.1.1: Use the wavelet-like decomposition module to obtain the characteristics of different frequency bands: The foreground information obtained in 2.3 is decomposed into high-frequency information F using a wavelet-like decomposition module. HH , high and low frequency information F HL , low and high frequency information F LH and low frequency information F LL , the process is shown in formula (5): FAVORITE HH ,FAVORITE HL ,FAVORITE LH ,FAVORITE LL =LWD(f i '),i∈{1,2,3,4} (5) Where LWD(·) represents a wavelet-like decomposition operation, f i ' is the foreground information obtained by separating the foreground and background; Step 3.1.2: Strengthen high-frequency information: The low-frequency information, high-low-frequency information, and high-frequency information are convolved by 1×1 and 3×3 respectively to obtain the feature F' LH 、F' HL and F' HH , then flatten the feature size and adjust the size to hw×c; perform matrix multiplication operation on the adjusted low-frequency and high-frequency features, that is, (F′ LH ) Τ ×F′ HL , and then perform sigmoid operation to obtain the correlation graph F cor , whose dimension is c×c, and its process is expressed as formula (6): F cor =Sigmoid((F L ' H ) Τ ×F H ' L ) (6) Finally, the obtained correlation graph is matrix multiplied with the adjusted high and high frequency feature F', and then adjusted back to the original tensor size, HH Then, the feature Fc obtained by splicing the three high-frequency information and processing it with 1×1 convolution is concatenated, and then the fused high-frequency feature F is obtained by 1×1 convolution. H , the process is expressed as formula (7)-(8): F c =conv(concat{F LH ,F HL ,F HH }) (7) F H =conv(concat((Reshape(Reshape(F H ' H )×F cor ),F c ))) (8) Among them, concat(·) represents the channel concatenation operation, conv(·) represents the 1×1 convolution operation, and Reshape(·) represents the tensor adjustment operation; Step 3.1.3: for the high frequency feature F H To make corrections: The low frequency information F LL Using low frequency features F L Indicates that the low-frequency feature F L And the obtained high frequency feature F H After 1×1 and 3×3 convolution operations respectively and adjusting the feature tensor size to hw×c, we get F l 、F h , calculate the correlation between the two, extract the features with high correlation, select the most relevant feature S, and then compare it with F h Perform matrix multiplication, and then adjust the tensor and add it to the high-frequency feature F H Concatenate to get feature F R , and finally obtain the corrected feature Fr through a residual structure; The residual structure includes two branches, one branch uses 1×1 convolution and Sigmoid (·) function to calculate the attention map; The other branch performs consecutive 1×1 and 3×3 convolutions; then, the outputs of the two branches are multiplied to obtain the modified feature F r ; Step 3.1.4, calculate the correlation: The correlation calculation module includes using the Pierre coefficient to perform autocorrelation calculation, a percentage pooling layer, and selecting Top k , get the processed foreground feature f i ; The process is expressed as formula (14): f i =Top k (PP(SC(F r ))) (14) Among them, PP(·) represents the percentage pooling layer, and SC(·) represents the autocorrelation calculation; Step 3.2: Process the background information and stitch the obtained foreground and background stitching features: The background processing module also includes a wavelet-like decomposition, a high-frequency fusion module, and a frequency domain correction module. The output of the frequency domain correction module is used as the output of the background processing module to obtain the processed background information b i ; Finally, the obtained foreground information and background information are spliced ​​to obtain the foreground and background spliced ​​features F1, F2, F3 and F4; The fourth step is to process the features F1 and F4 after the foreground and background are spliced ​​through the edge auxiliary branch to obtain the predicted edge Step 4.1: The features F1 and F4 after the foreground and background are concatenated are respectively subjected to the Sobel operator to extract the rough edges F1' and F'4; In step 4.2, the attention mechanism is used to refine the edge features. The process is expressed as formula (16)-(17): F sc =conv(concat(F sp ,F ch )) (17) Among them, SAM(·) represents the spatial attention mechanism, and CAM(·) represents the channel attention mechanism; Step 4.3: The final predicted edge P is obtained by formula (18): edge : P edge =conv(concat(F sc ,F1',F4)) (18) The fifth step is to use the refinement fusion module to fuse the multi-scale features and obtain the final prediction results. The different colors in the prediction results represent untampered, spliced ​​tampered, copy-paste tampered, and removed tampered: F1, F2 and F3 are respectively compared with the predicted edge P edge Multiply them to get the corrected features E1, E2, and E3. The corrected features E1, E2, and E3 are processed by the refinement and fusion module and the output features are recorded as P1, P2, and P3 respectively. The corrected features are negated to get the negated corrected features 1-E1, 1-E2, and 1-E3. The corrected features E1, E2, and E3 and the corresponding negated corrected features 1-E1, 1-E2, and 1-E3 are input into the three refinement and fusion modules. The refinement fusion module includes three parallel refinement modules and three fusion modules. The three input features of the refinement fusion module are processed by a refinement module respectively to obtain enhanced features F3 and F4. r 、F1 r and F2 r ; F1 r and F2 r After being processed by the first fusion module, With F3 r After being processed by the second fusion module, After being processed by the third fusion module, the output feature P of the current refinement fusion module is obtained. i ; The output P1 of the first refinement fusion module after 1×1 convolution and sigmoid function processing is used as the prediction result of the final model; The refinement module includes three Conv-BN-ReLUs layers. The input is first enhanced by the first Conv-BN-ReLUs layer to obtain the feature H i Then, in order to highlight the blurred area, H i With 1-H i Element-wise multiplication is performed, and the value of the blurred area will be large by element-wise multiplication, which can well highlight the blurred area; then it is processed by the second conv-BN-ReLUs layer and element-wise multiplication with the input, and then concatenated with the input, and finally processed by the third conv-BN-ReLUs layer to obtain the enhanced feature F i r ; The fusion module includes a concatenation operation, a Conv-BN-ReLUs layer and a sigmoid function. The process of the three fusion modules is expressed as formulas (22)-(23): Where Conv(·) represents the Conv-BN-ReLUs layer; Step 6: Supervise the prediction results Step 6.1: Apply binary cross entropy loss L BCE To supervise the predicted mask, it is expressed as formula (24): in, represents the truth value, and P i x,y Represents the prediction result of the tampered image; l∈{0,1,2,3} represents different tampering categories: real, splicing tampering, copy-paste tampering and removal tampering; Pr represents the probability; Step 6.2, apply L dice To detect the predicted edge, it is expressed as formula (25): in, Represents the prediction result of each pixel in the prediction edge, Represents the true value of each pixel, H and W are the height and width of the image; The total loss during training is the marginal prediction loss L dice and the mask prediction loss L BCE sum; A multi-operation image dataset is used to train a multi-tampering localization model, and the trained multi-tampering localization model is used to locate different types of tampered areas in the image.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of any method described in claims 1-5 can be implemented.

Images