A traditional image forgery positioning device and method based on multi-scale attention

Abstract

The application relates to an image forgery positioning device and method based on multi-scale attention, and belongs to the technical field of image processing, solving the problems that the edge of a positioning area is not clear enough and the positioning result is not accurate enough in the prior art. The application generates synthetic forged images of various types and modes through a traditional forged image synthesis module, enhancing the generalization of the device to various types of forgery. The forged area edge feature extraction module extracts and strengthens the part related to the forged area edge in the forged area feature, fully utilizes the forged traces brought by the edge operation, and has high edge definition. The multi-scale spatial attention module gradually fuses spatial features of different scales, provides coarse-grained forged area prediction from small-scale features, combines large-scale features and edge features for fine-grained forged area prediction, and realizes more precise forged area positioning. The device training is easy to control and sufficient.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a traditional image forgery localization device and method based on multi-scale attention. Background Technology

[0002] Image forgery techniques include splicing, copying and moving, erasing, and retouching. With the support of image editing methods and image retouching software, these forgeries have become much easier to obtain, and more and more fake images have appeared on the Internet. Among these fake images are malicious forgeries that smear individuals, businesses, and society, and deliberately provide false information, which has a negative impact on society.

[0003] Currently, several image forgery detection methods exist to detect and identify forgery methods such as splicing, copying and moving, erasing, and retouching. Generally, existing methods mostly focus on the inconsistencies between the forged area and the real area, ignoring the edges where the forged area meets the real area. However, due to common forgery operations such as blurring and Poisson blending at splicing edges, these edges often contain more forgery traces. Therefore, while these methods can roughly locate the forged area, they sometimes suffer from unclear edges and inaccurate localization results.

[0004] In summary, existing image forgery detection technologies suffer from problems such as unclear edges of the located areas and inaccurate localization results. Summary of the Invention

[0005] In view of the above problems, the present invention provides a traditional image forgery localization device and method based on multi-scale attention, which solves the problems of unclear edges of the localized area and inaccurate localization results in the prior art.

[0006] This invention provides a traditional image forgery localization device based on multi-scale attention, comprising: a traditional forged image synthesis module, a forged region feature extraction module, a forged region edge feature extraction module, a multi-scale spatial attention module, a mask output module, and a forged region mask thinning module, wherein...

[0007] The traditional forged image synthesis module is used to process real images to obtain traditional forged images, and then combine them with pre-synthesized images to obtain a training set;

[0008] The forged region feature extraction module is used to encode and decode the training set to obtain forged region features;

[0009] The forged region edge feature extraction module is used to perform edge information extraction processing on the forged region features to obtain the forged region edge features;

[0010] The multi-scale spatial attention module is used to calculate the relationship between the fake region features at two scales based on the fake region features, adjust the fake region features at the larger scale of the two scales, and combine them with the fake region edge features at the same scale as the fake region features at the larger scale to calculate the attention matrix.

[0011] The mask output module and the forgery region mask refinement module are used to process the attention matrix to obtain the final image mask and complete the image forgery localization.

[0012] Furthermore, the forged region feature extraction module employs a convolutional neural network with an encoder-decoder structure to extract forged region features. The encoder is used to perform preliminary extraction of forged region features, obtaining global and local features. The decoder is used to distinguish between forged regions and background regions in the features at various scales, and gradually fuses the global and local features to obtain the forged region features.

[0013] Furthermore, the fake region edge feature extraction module consists of several fake region edge feature extraction sub-modules, each having a convolutional layer, a batch normalization layer, and a ReLU activation function. Each fake region edge feature extraction sub-module is used to extract corresponding edge features from the fake region features to obtain fake region edge features.

[0014] Furthermore, the mask output module consists of several mask output sub-modules with one convolutional layer and one upsampling layer; the mask output sub-module is used to reduce the dimension of the attention matrix output by the multi-scale spatial attention module to a single channel and upsample it to the size of the input image to obtain an intermediate image mask, which serves as the intermediate output of the image forgery localization device. Its spatial size is 1×H×W, representing the number of channels, height and width of the intermediate output, respectively.

[0015] Furthermore, the forged region mask refinement module is a convolutional neural network, which uses a pooling pyramid to refine the intermediate image mask output by the mask output submodule to obtain the final image mask, thereby realizing the localization of the forged region.

[0016] The term "traditional" in the traditional image forgery synthesis module refers to manually using Photoshop or other image editing software to forge images.

[0017] This invention also provides a traditional image forgery localization method based on multi-scale attention, implemented using the aforementioned traditional image forgery localization device based on multi-scale attention, comprising the following steps:

[0018] S1: Construct a real image dataset;

[0019] S2: Input the real image dataset into the traditional fake image synthesis module to generate fake images, and then combine them with the pre-synthesized images in PSCC-Net to obtain the training set;

[0020] S3: Input the images in the training set into the forgery region feature extraction module, and after encoding and decoding, calculate several forgery region features of n scales corresponding to the images;

[0021] S4: Input the several fake region features at the n scales into the fake region edge feature extraction module to calculate the several fake region edge features at the n scales;

[0022] S5: Input several fake region features and several fake region edge features at the n scales into the multi-scale spatial attention module to calculate the attention matrix at the n scales;

[0023] S6: Input the attention matrices of the n scales into the mask output module to calculate the intermediate image masks out1, out2, ..., out n ;

[0024] S7: Input the out1 into the fake region mask thinning module, and use the pooling pyramid to thin out1 to obtain the final image mask out0;

[0025] S8: Repeat steps S3 to S7 until the hybrid loss function converges, completing the training phase and obtaining a trained image forgery localization device based on multi-scale attention.

[0026] S9: In the usage phase, using the image forgery localization device based on multi-scale attention trained in step S8, steps S3 to S7 are executed on the test image to obtain out0 corresponding to the test image, thus completing the localization of the forgery region of the test image.

[0027] Furthermore, the step of inputting the real image dataset into the traditional forged image synthesis module to generate traditional forged images, and then combining them with the pre-synthesized images in PSCC-Net to obtain a training set, including:

[0028] S2-1: Select a real image from the real image dataset as the background image I. bg And randomly select another image from the real dataset as the foreground image I. fg , will I bg and I fg Cut to the same size H×W; where H refers to height and W refers to width;

[0029] S2-2: For I bg and Ifg Perform compression or no compression on the random quality factor, wherein the random quality factor ranges from 35 to 100, and preset the probability of no compression;

[0030] S2-3: Randomly select a random graphic mask or a pre-trained target segmentation network as the mask generator, and use the I obtained in step S2-2... fg Input the mask generator to obtain the mask M; where M has a dimension of H×W×1 and the value range of all elements in M ​​is {0, 1};

[0031] S2-4: The I obtained in step S2-2 fg Perform the same translation on M obtained in steps S2-3, moving the center of the region in M ​​with an element value of 1 to the center of the image. Then, perform the same random affine transformation, where the probability of rotation is preset, the rotation angle range is [-180°, 180°], and the probability of scaling is preset, the scaling factor range is [0.2, 3]. Finally, perform the same random affine transformation on I. fg Perform a random translation on M, moving the region in M ​​with an element value of 1 to a random position within the image;

[0032] S2-5: I obtained through step S2-2 bg The I obtained in step S2-4 fg And M, the traditional forged image I is calculated in the following way. f :I f =I bg ×(1-M)+I fg ×M;

[0033] S2-6: Repeat steps S2-1 to S2-5 to obtain a large number of traditional forged images to form a forged image set, and then combine it with the pre-synthesized images in PSCC-Net to obtain a training set; wherein, the spatial size of the traditional forged image is C×H×W, and C represents the number of channels of the image features.

[0034] Furthermore, the aforementioned "several forged region features at n scales" refers to seven forged region features at six scales, namely: h1: H1×W1×64, h2: H2×W2×128, h3: H3×W3×256, h4: H4×W4×512, h5: H5×W5×512, h6: H6×W6×512, h7: H7×W7×512, where H1 = H, W1 = W.

[0035] The step of inputting several fake region features at n scales into the fake region edge feature extraction module to calculate several fake region edge features at n scales refers to inputting the seven fake region features into seven sub-modules of the fake region edge feature extraction module respectively to obtain seven fake region edge features e1: H1×W1×64, e2: H2×W2×128, e3: H3×W3×256, e4: H4×W4×512, e5: H5×W5×512, e6: H6×W6×512, e7: H7×W7×512.

[0036] Furthermore, several fake region features and several fake region edge features at n scales are input into a multi-scale spatial attention module to calculate attention matrices at n scales. Specifically, h1, h2, h3, h4, h5, h6, h7 and e1, e2, e3, e4, e5, e6, e7 are input into the multi-scale spatial attention module to output attention matrices at six scales, namely m1, m2, m3, m4, m5, and m6. This process includes the following steps:

[0037] S5-1: Reduce the dimensionality of h7 through a fully connected layer to... Obtain h′7 and denote it as m7;

[0038] S5-2: m i+1 h i and e i As the input to the i-th spatial attention submodule, where i = 1, 2, 3, 4, 5, 6; the i-th spatial attention submodule first passes h through a fully connected layer i Dimensionality reduction to H i ×W i ×1, denoted as h′ i Calculate z i =h′ i (m i+1 ) T Then for z i After performing the softmax operation, calculate z′. i =z i (h′ i Finally, a fully connected layer is used to apply z′. i Perform a linear transformation and calculate m. i =z′ i +e i , which serves as the output of the i-th spatial attention submodule.

[0039] Furthermore, the hybrid loss function is expressed as:

[0040]

[0041] Where, α k For l (k) The weight, l (k) out k The resulting loss is expressed as:

[0042]

[0043] in, The binary entropy loss function is expressed as:

[0044]

[0045] Where G(r, c)∈{0,1} is the label at position (r, c) in the image, out k (r, c)∈[0, 1] is the device's prediction of the position (r, c) in the image;

[0046] in, Represented as:

[0047]

[0048] Where x and y are N×N image patches cut from the same position at G and out, respectively, and μ x μ y and σ x , σ y The mean and standard deviation of x and y are respectively, and σ is the standard deviation of x and y. xy Let C1 be the covariance of x and y, and C1 = 0.01. 2 And C2 = 0.03 2 Used to avoid the case where the denominator of a division is 0;

[0049] in, Represented as:

[0050]

[0051] Where G(r, c)∈{0,1} is the label at position (r, c) in the image, out k (r, c)∈[0,1] is the device's prediction of the position (r, c) in the image.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects:

[0053] (1) The traditional image forgery localization device and method based on multi-scale attention of the present invention can generate more diverse and varied synthetic forgery images through the traditional forgery image synthesis module, thereby enhancing the generalization of the device for various types of forgery.

[0054] (2) The traditional image forgery localization device and method based on multi-scale attention of the present invention extracts and enhances the part of the features obtained by the forgery region feature extraction module that is related to the edge of the forgery region through the forgery region edge feature extraction module, making full use of the forgery traces brought about by edge blurring and other operations, and the localized region edge has high clarity.

[0055] (3) The traditional image forgery localization device and method based on multi-scale attention of the present invention integrates spatial features obtained at different scales step by step through the multi-scale spatial attention module. The smaller scale features provide coarse-grained prediction of the forgery region, and then the larger scale features and edge features are combined to perform finer-grained prediction of the forgery region, thereby achieving more refined forgery region localization.

[0056] (4) The traditional image forgery localization device and method based on multi-scale attention of the present invention are easy to control during the device training process and the device training is sufficient. Attached Figure Description

[0057] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0058] Figure 1 This is a schematic diagram of a conventional image forgery localization device based on multi-scale attention according to an embodiment of the present invention.

[0059] Figure 2 This is a flowchart illustrating the training process of a traditional image forgery localization method based on multi-scale attention according to an embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram of the structure and process of the traditional forged image synthesis module in a traditional image forgery localization device based on multi-scale attention according to an embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram of the structure and process of a multi-scale spatial attention module in a traditional image forgery localization device based on multi-scale attention according to an embodiment of the present invention.

[0062] Figure 5 This is a schematic diagram illustrating the usage process of a traditional image forgery localization method based on multi-scale attention according to an embodiment of the present invention. Detailed Implementation

[0063] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0064] Because the features required for image forgery detection are often related to the image acquisition, encoding, and forgery processes but not to the semantic content of the image, these signals are very weak and are often masked by semantic information or destroyed by compression, scaling, blurring, and other processes during image transmission. Furthermore, in fields such as image classification and object detection, devices focus more on the semantic content of the image and are less affected by image scaling operations. Therefore, images are usually scaled to a smaller size, such as 224×224, to avoid memory overflow caused by large image sizes. However, for image forgery detection, scaling the input image can destroy the weak signals related to forgery, weakening the device's performance. This makes weak signal extraction in image forgery detection and localization research one of the research challenges. How to construct an effective feature extraction device that takes into account both global features and local weak signals, is more robust to image scaling and compression operations, and can fully extract image forgery features is a key research issue.

[0065] Example 1

[0066] A specific embodiment of the present invention, such as Figure 1 As shown, a traditional image forgery localization device based on multi-scale attention is disclosed, comprising: a traditional forged image synthesis module, a forged region feature extraction module, a forged region edge feature extraction module, a multi-scale spatial attention module, a mask output module, and a forged region mask thinning module, wherein,

[0067] The traditional forged image synthesis module is used to process real images to obtain traditional forged images, which are then combined with pre-synthesized images to obtain a training set;

[0068] The fake region feature extraction module is used to encode and decode the training set to obtain fake region features;

[0069] The forged region edge feature extraction module is used to extract edge information from the forged region features to obtain the forged region edge features;

[0070] The multi-scale spatial attention module is used to calculate the relationship between fake region features at two scales based on fake region features. It adjusts the fake region features at the larger scale of the two scales and combines them with the fake region edge features at the same scale to calculate the attention matrix.

[0071] Optionally, the pre-synthesized image is a pre-synthesized image in PSCC-Net.

[0072] Specifically, such as Figure 3 As shown, the traditional fake image synthesis module uses real images for synthesis to obtain traditional fake images, which are then combined with pre-synthesized images in PSCC-Net to obtain a training set. Two real, un-fake images are used as the foreground and background, respectively. A pre-trained target segmentation network and random image generation method are used to generate masks for the fake regions. The corresponding regions in the foreground are randomly compressed, scaled, rotated, and translated using JPEG, and then stitched into the background image as fake regions. For example, JPEG compression is used. Then, random fusion methods, including direct stitching, Poisson fusion, alpha fusion, and Gaussian blur, are applied to synthesize the fake image. The spatial dimensions of the synthesized fake image are C×H×W, representing the number of channels, height, and width of the image features, respectively. Here, C represents the channel dimension, and H and W represent the spatial dimensions. "Traditional" refers to image forgery performed manually using Photoshop or other image editing software, as opposed to image forgery using deep learning methods.

[0073] The forged region feature extraction module uses a convolutional neural network with an encoder-decoder structure to extract forged region features. The encoder is used to perform preliminary extraction of forged region features, obtaining global and local features. The decoder is used to distinguish forged regions and background regions in the features at various scales, and gradually fuses global and local features to obtain forged region features.

[0074] The fake region edge feature extraction module consists of several fake region edge feature extraction sub-modules, each with a convolutional layer, a batch normalization layer, and a ReLU activation function. Each fake region edge feature extraction sub-module is used to extract the corresponding edge features from the fake region features to obtain the fake region edge features.

[0075] like Figure 4 As shown, the multi-scale spatial attention module is used to calculate the correlation between the predicted mask of the fake region at each scale and the predicted mask of the fake region at a smaller scale based on the features of the fake region. This relationship is used as spatial attention to fine-tune the predicted mask at that scale and combine it with the edge features of the fake region at the corresponding scale to more accurately locate the fake region.

[0076] The mask output module consists of several mask output sub-modules with one convolutional layer and one upsampling layer. The mask output sub-module is used to reduce the dimension of the attention matrix output by the multi-scale spatial attention module to a single channel and upsample it to the size of the input image to obtain an intermediate image mask, which serves as the intermediate output of the image forgery localization device. Its spatial size is 1×H×W, representing the number of channels, height and width of the intermediate output, respectively.

[0077] The fake region mask refinement module is a convolutional neural network. It uses a pooling pyramid to refine the intermediate image mask output by the mask output submodule to obtain the final image mask, thereby locating the fake region.

[0078] In an optional implementation, the forged region feature extraction module is BASNet, which uses a convolutional neural network. The input image size is 256×256 pixels, and the decoder can obtain h1: 256×256×64, h2: 128×128×128, h3: 64×64×256, h4: 32×32×512, h5: 16×16×512, h6: 8×8×512, h... bg Seven fake region features at six scales: 8×8×512.

[0079] In an optional implementation, the fake region edge feature extraction module uses a convolutional neural network and includes seven sub-modules, which extract the corresponding fake region edge features from the seven features extracted by the decoder of the fake region feature extraction module.

[0080] In an optional implementation, the multi-scale spatial attention module is divided into six sub-modules, which respectively reduce and fuse the features at the six scales and their adjacent scales. The input to each sub-module is the attention matrix output by the previous scale spatial attention module, the forged region features extracted at that scale, and the forged region edge features extracted at that scale. A fully connected layer then converts the features extracted by the forged region feature extraction module and the forged region edge feature extraction module into a shape C. i ×H i ×W i Dimensionality reduced to 1×H i ×W i The values ​​i = 1, 2, 3, 4, 5, 6 are fused using matrix multiplication and addition to obtain the spatial attention at this scale, which serves as one of the inputs to the spatial attention module at the next scale. The number of input feature channels C corresponding to the six sub-modules is... i Let i = 1, 2, 3, 4, 5, 6 be C6 = 512, C5 = 512, C4 = 512, C3 = 256, C2 = 128, and C1 = 64 respectively, and let H be the length of the feature. iLet i = 1, 2, 3, 4, 5, 6, where H1 = 256, H2 = 128, H3 = 64, H4 = 32, H5 = 16, H6 = 8 respectively, and the feature width W is... i The numbers i = 1, 2, 3, 4, 5, 6 are respectively W1 = 256, W2 = 128, W3 = 64, W4 = 32, W5 = 16, and W6 = 8.

[0081] In an optional implementation, the mask output module consists of six sub-modules, each with a convolutional layer and an upsampling layer.

[0082] In an optional implementation, the fake region masking refinement module is a convolutional neural network with nine convolutional layers.

[0083] Example 2

[0084] This invention also provides a traditional image forgery localization method based on multi-scale attention, implemented based on the traditional image forgery localization device based on multi-scale attention proposed in Embodiment 1, including the following steps, wherein steps S1 to S8 are the training phase and step S9 is the usage phase:

[0085] S1: Construct a real image dataset;

[0086] S2: Input the real image dataset into the traditional fake image synthesis module to generate fake images, and then combine them with the pre-synthesized images in PSCC-Net to obtain the training set; for example... Figure 2 As shown, step S2 specifically includes the following steps:

[0087] S2-1: Select a real image from the real image dataset as the background image I. bg And randomly select another image from the real dataset as the foreground image I fg , will I bg and I fg Cut to the same size H×W; where H refers to the height and W refers to the width.

[0088] S2-2: For I bg and I fg Perform compression or no compression on the random quality factor, where the random quality factor ranges from 35 to 100, and preset the probability of no compression. For example, the preset probability of no compression is 3 / 10.

[0089] S2-3: Randomly select a random graphic mask or a pre-trained target segmentation network as the mask generator, and use the I obtained in step S2-2... fg Input a mask generator to obtain a mask M; where M has dimensions H×W×1 and the value range of all elements in M ​​is {0, 1}.

[0090] S2-4: The I obtained in step S2-2 fg Perform the same translation as M obtained in steps S2-3, moving the center of the region in M ​​with an element value of 1 to the center of the image. Then perform the same random affine transformation, where the probability of rotation is preset (e.g., 2 / 3) and the rotation angle range is [-180°, 180°). The probability of scaling is also preset (e.g., 2 / 3) and the scaling factor range is [0.2, 3]. Finally, perform the same transformation on I. fg Perform a random translation on M, moving the region in M ​​with an element value of 1 to a random position within the image.

[0091] S2-5: I obtained through step S2-2 bg The I obtained in step S2-4 fg And M, the traditional forged image I is calculated in the following way. f :I f =I bg ×(1-M)+I fg ×M;

[0092] S2-6: Repeat steps S2-1 to S2-5 to obtain a large number of traditional forged images to form a forged image set, and then combine it with the pre-synthesized images in PSCC-Net to obtain the training set; wherein, the spatial size of the traditional forged image is C×H×W, and C represents the number of channels of the image features.

[0093] S3: Input the images in the training set into the fake region feature extraction module. After encoding and decoding, calculate several fake region features of n scales corresponding to the image.

[0094] Optionally, the synthetic forged data obtained in step S2 is input into the forged region feature extraction module, and the decoding process yields seven forged region features at six scales, namely h1: H1×W1×64, h2: H2×W2×128, h3: H3×W3×256, h4: H4×W4×512, h5: H5×W5×512, h6: H6×W6×512, and h7: H7×W7×512, where H1 = H and W1 = W.

[0095] S4: Input several fake region features at n scales into the fake region edge feature extraction module to calculate several fake region edge features at n scales.

[0096] Optionally, the seven features obtained in step S3 are input into the seven sub-modules of the fake region edge feature extraction module to obtain seven fake region edge features e1: H1×W1×64, e2: H2×W2×128, e3: H3×W3×256, e4: H4×W4×512, e5: H5×W5×512, e6: H6×W6×512, and e7: H7×W7×512. The seven fake region edge features are then reduced to a single channel through a single-layer convolutional network and used as the edge mask output of the device for calculating the loss function.

[0097] S5: Input several fake region features and several fake region edge features at n scales into the multi-scale spatial attention module to calculate the attention matrix at n scales.

[0098] Optionally, h1, h2, h3, h4, h5, h6, h7 obtained in step S3 and e1, e2, e3, e4, e5, e6, e7 obtained in step S4 are input into the multi-scale spatial attention module, which outputs attention matrices of six scales, namely m1, m2, m3, m4, m5, and m6, respectively. The specific steps include:

[0099] S5-1: Reduce the dimensionality of h7 using a fully connected layer to... Obtain h′7 and denote it as m7.

[0100] S5-2: m i+1 h i and e i As the input to the i-th spatial attention submodule, where i = 1, 2, 3, 4, 5, 6; the i-th spatial attention submodule first passes h through a fully connected layer i Dimensionality reduction to H i ×W i ×1, denoted as h′ i Calculate z i =h′ i (m i+1 ) T Then for z i After performing the softmax operation, calculate z′. i =z i (h′ i Finally, a fully connected layer is used to apply z′. i Perform a linear transformation and calculate m. i =z′ i +e i , which serves as the output of the i-th spatial attention submodule.

[0101] S6: Input the attention matrices of n scales into the mask output module to calculate the intermediate image masks out1, out2, ..., outn .

[0102] Optionally, the attention matrices m1, m2, m3, m4, m5, and m6 output in step S5 are input into the mask output module, convolved, and upsampled to the size of the input image to serve as intermediate outputs out1, out2, out3, out4, out5, and out6 of the device.

[0103] S7: Input out1 into the fake region mask thinning module, use pooling pyramid to thin out1, and obtain the final image mask out0.

[0104] Optionally, the out1 obtained in step S6 is input into the fake region mask refinement module, and the pooling pyramid is used to refine out1 to obtain the final positioning result out0 of the device.

[0105] S8: Repeat steps S3 to S7 until the hybrid loss function converges, completing the training phase and obtaining a trained image forgery localization device based on multi-scale attention.

[0106] Specifically, all parameters in the forgery region feature extraction module in step S3, the forgery region edge feature extraction module in step S4, the multi-scale spatial attention module in step S5, the mask output module in step S6, and the forgery region mask refinement module in step S7 are saved to obtain the trained image forgery localization device based on multi-scale attention.

[0107] S9: In the usage phase, using the image forgery localization device based on multi-scale attention trained in step S8, steps S3 to S7 are executed on the test image to obtain out0 corresponding to the test image, thus completing the localization of the forgery area of ​​the test image.

[0108] Optionally, during the usage phase, the image forgery localization device trained in step S8, using a test set as input data, is used to locate the forged regions. Steps S3 to S7 are executed for all input data, with step S7 obtaining the final localization result out0 and calculating relevant metrics.

[0109] like Figure 5 As shown, the usage process of a traditional image forgery localization method based on multi-scale spatial attention provided according to an embodiment of the present invention will be described in detail. Figure 5 This is a schematic diagram of the usage process of the image forgery localization method based on multi-scale attention according to an embodiment of the present invention; when using the image forgery localization method based on multi-scale attention, step S9 is performed to obtain the localization mask out0∈[0,1] of the input image, where the region with a value of 1 represents the forgery region and the region with a value of 0 represents the real region.

[0110] In an optional implementation, the pre-trained target segmentation network in steps S2-3 is RefineNet, which is pre-trained using data from the VOC dataset with target region mask labels.

[0111] Optionally, the process of locating the fake region in step S7 is obtained through a hybrid loss function, which is expressed as:

[0112]

[0113] Where, α k For l (k) The weight, l (k) out k The resulting loss is expressed as:

[0114]

[0115] in, The binary entropy loss function is expressed as:

[0116]

[0117] Where G(r, c)∈{0,1} is the label at position (r, c) in the image, out k (r, c)∈[0,1] is the device's prediction of the position (r, c) in the image.

[0118] in, Represented as:

[0119]

[0120] Where x and y are N×N image patches cut from the same position at G and out, respectively, and μ x μ y and σ x , σ y The mean and standard deviation of x and y are respectively, and σ is the standard deviation of x and y. xy Let C1 be the covariance of x and y, and C1 = 0.01. 2 And C2 = 0.03 2 This is used to avoid the case where the denominator of a division is 0.

[0121] in, Represented as:

[0122]

[0123] Where G(r, c)∈{0,1} is the label at position (r, c) in the image, out k(r, c)∈[0,1] is the device's prediction of the position (r, c) in the image.

[0124] Optionally, the number of iterations in step S8 can be set to t = 40 iterations to achieve better results and reduce computational load. In other implementations, this number of iterations can be increased or decreased as needed.

[0125] Optionally, in step S8, the amount of training data used in each iteration is 25,000, the amount of validation data is 600, and the batch size can be set to batch=6. In other implementations, the number of rounds can be increased or decreased as needed.

[0126] The setup in the above-described method provided by the embodiments of the present invention has the advantages of easy control of the device training process and sufficient device training.

[0127] The following details the training process of the image forgery localization device based on multi-scale attention provided according to an embodiment of the present invention. Using training data from PSCC-Net comprising 376,739 images and forged images synthesized by a conventional forged image synthesis module comprising 70,000 images as the training set, the device is trained according to steps S3-S8 described above. Adam (Adaptive Momentum Estimation Optimization Algorithm) is used as the optimizer for the device, with a learning rate set to 0.0003. The amount of training data used in each iteration is 25,000, the amount of validation data is 600, and the batch size is set to batch = 6. A total of 40 iterations are performed, with the learning rate decreasing to 0.3 every 10 iterations. Finally, the device with the best validation set evaluation metric is saved as the final device.

[0128] During the device usage phase, a multi-scale attention-based image forgery localization method was used to locate forged regions on test sets such as IMD20, Columbia, and Casia datasets, which contained 180, 6044, and 2010 forged images, respectively. First, the trained device is loaded, and the fake regions are located according to step S9 above. The results are compared with the real labels. On the IMD20 dataset, the method's AUC (Area Area of ​​the Receiver Curve) reaches 84.8%, surpassing the 2022 state-of-the-art (SOTA) paper ObjectFormer's AUC of 82.1% on the same dataset. On the Columbia dataset, the method's AUC reaches 97.2%, surpassing ObjectFormer's AUC of 95.5% on the same dataset. On the Casia dataset, the method's AUC reaches 83.6%, slightly inferior to ObjectFormer's AUC of 84.3% on the same dataset. However, the finely tuned device on the Casia2 dataset achieves an AUC of 88.4% on the Casia1 dataset, outperforming ObjectFormer's finely tuned device's AUC of 88.2% under the same conditions. The data demonstrate that the method provided by the embodiments of the present invention yields high-level results, illustrating the efficiency and feasibility of the present invention.

[0129] Compared with existing technologies, the traditional image forgery localization device and method based on multi-scale attention of the present invention can generate more diverse and varied synthetic forgery images through the traditional forgery image synthesis module, enhancing the device's generalization ability to various types of forgery; the forgery region edge feature extraction module extracts and enhances the parts related to the forgery region edge from the features obtained by the forgery region feature extraction module, making full use of the forgery traces brought by edge blurring and other operations, resulting in high clarity of the located region edge; the multi-scale spatial attention module fuses spatial features obtained at different scales step by step, with smaller-scale features providing coarse-grained forgery region prediction, and then combining larger-scale features and edge features to perform finer-grained forgery region prediction, achieving more precise forgery region localization; the device training process is easy to control and the device training is sufficient.

[0130] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A traditional image forgery localization device based on multi-scale attention, characterized in that, include: The traditional forged image synthesis module includes a forged region feature extraction module, a forged region edge feature extraction module, a multi-scale spatial attention module, a mask output module, and a forged region mask refinement module. The traditional forged image synthesis module is used to process real images to obtain traditional forged images, and then combine them with pre-synthesized images to obtain a training set; The forged region feature extraction module is used to encode and decode the training set to obtain forged region features; The forged region edge feature extraction module is used to perform edge information extraction processing on the forged region features to obtain the forged region edge features; The multi-scale spatial attention module is used to calculate the relationship between the fake region features at two scales based on the fake region features, adjust the fake region features at the larger scale of the two scales, and combine them with the fake region edge features at the same scale as the fake region features at the larger scale to calculate the attention matrix. The mask output module and the forgery region mask refinement module are used to process the attention matrix to obtain the final image mask and complete the image forgery localization. The forged region feature extraction module uses a convolutional neural network with an encoder-decoder structure to extract forged region features. The encoder is used to perform preliminary extraction of forged region features to obtain global and local features. The decoder is used to distinguish forged regions and background regions in the features at various scales and gradually fuse the global and local features to obtain the forged region features. The fake region edge feature extraction module consists of several fake region edge feature extraction sub-modules, each with a convolutional layer, a batch normalization layer, and a ReLU activation function. Each fake region edge feature extraction sub-module is used to extract corresponding edge features from the fake region features to obtain fake region edge features.

2. The traditional image forgery localization device based on multi-scale attention according to claim 1, characterized in that, The mask output module consists of several mask output sub-modules, each with a convolutional layer and an upsampling layer. These sub-modules reduce the dimensionality of the attention matrix output by the multi-scale spatial attention module to a single channel and upsample it to the size of the input image to obtain an intermediate image mask. This mask serves as the intermediate output of the image forgery localization device, and its spatial size is 1× H × W , representing the number of intermediate output channels, height, and width, respectively.

3. The traditional image forgery localization device based on multi-scale attention according to claim 2, characterized in that, The forged region mask refinement module is a convolutional neural network. It uses a pooling pyramid to refine the intermediate image mask output by the mask output submodule to obtain the final image mask, thereby locating the forged region. The term "traditional" in the traditional image forgery synthesis module refers to manually using Photoshop or other image editing software to forge images.

4. A traditional image forgery localization method based on multi-scale attention, characterized in that, The method, based on any one of claims 1 to 3, utilizes a conventional image forgery localization device based on multi-scale attention, comprising the following steps. S1: Construct a real image dataset; S2: Input the real image dataset into the traditional fake image synthesis module to generate fake images, and then combine them with the pre-synthesized images in PSCC-Net to obtain the training set; S3: Input the images in the training set into the forgery region feature extraction module, and after encoding and decoding, calculate the corresponding features of the images. n Several forged regional features at various scales; S4: The above n Several forged region features at various scales are input into the forged region edge feature extraction module to calculate... n Several fake region edge features at various scales; S5: [The sentence is incomplete and likely refers to a specific item or clause.] n Several forged region features at various scales and the aforementioned n Several fake region edge features at various scales are input into a multi-scale spatial attention module to calculate... n Attention matrices of various scales; S6: [The sentence is incomplete and likely refers to a specific method or procedure.] n The attention matrix at various scales is input to the mask output module, which calculates the intermediate image mask. ; S7: [The sentence is incomplete and likely refers to a specific item or instruction.] Input forgery region mask refinement module, using pooled pyramid pair Refine the image to obtain the final image mask. ; S8: Repeat steps S3 to S7 until the hybrid loss function converges, completing the training phase and obtaining a trained image forgery localization device based on multi-scale attention. S9: In the usage phase, using the image forgery localization device trained in step S8 based on multi-scale attention, steps S3 to S7 are executed on the test image to obtain the image corresponding to the test image. This completes the localization of the forged area in the test image.

5. The traditional image forgery localization method based on multi-scale attention according to claim 4, characterized in that, The process involves inputting the real image dataset into a traditional forged image synthesis module to generate traditional forged images, which are then combined with pre-synthesized images in PSCC-Net to obtain a training set, including: S2-1: Select a real image from the real image dataset as the background image. And randomly select another image from the real dataset as the foreground image. ,Will and Cut to the same size ;in, H Refers to height, W Refers to width; S2-2: To and Perform compression or no compression on the random quality factor, wherein the random quality factor ranges from 35 to 100, and preset the probability of no compression; S2-3: Randomly select a random graphic mask or a pre-trained target segmentation network as the mask generator, and use the mask obtained in step S2-2... Input the mask generator to obtain the mask. ;in, The dimension is , The range of all elements in the set is ; S2-4: The result obtained in step S2-2 and obtained in step S2-3 Perform the same translation, The center of the region with an element value of 1 is moved to the center of the image, and then the same random affine transformation is performed, where the probability of rotation is preset, and the rotation angle range is... It also presets the probability of scaling, with the scaling factor ranging from [value missing]. Finally, for and Perform a random translation, and... The region with an element value of 1 is moved to a random position within the image; S2-5: Obtained through step S2-2 The results obtained in step S2-4 and Traditional forged images are calculated in the following way. : ; S2-6: Repeat steps S2-1 to S2-5 to obtain a large number of the traditional forged images to form a forged image set, and then combine it with the pre-synthesized images in PSCC-Net to obtain a training set; wherein, the spatial size of the traditional forged images is , C The number of channels representing image features.

6. The traditional image forgery localization method based on multi-scale attention according to claim 5, characterized in that, The n Several forged region features at six scales refer to seven forged region features at six scales, namely... , , , , , , ,in ; The n Several forged region features at various scales are input into the forged region edge feature extraction module to calculate... n Several fake region edge features at various scales refer to inputting the seven fake region features into seven sub-modules of the fake region edge feature extraction module to obtain seven fake region edge features. , , , , , , .

7. The traditional image forgery localization method based on multi-scale attention according to claim 6, characterized in that, The n Several forged region features at various scales and the aforementioned n Several fake region edge features at various scales are input into a multi-scale spatial attention module to calculate... n Attention matrix of a certain scale refers to the attention matrix of the aforementioned scale. and stated Input a multi-scale spatial attention module and output attention matrices at six scales, respectively. Specifically, it includes the following steps: S5-1: The above Dimensionality reduction through a fully connected layer ,get And recorded as ; S5-2: Will , and As the first The input of each spatial attention submodule, where Among them, the first The spatial attention submodule first passes through a fully connected layer... Dimensional reduction , recorded as ,calculate Then to Do After the operation, calculate Finally, a fully connected layer is used to... Perform linear transformations and calculate As the first The output of each spatial attention submodule.

8. The traditional image forgery localization method based on multi-scale attention according to claim 7, characterized in that, The hybrid loss function is expressed as: in, for The weight, For the reason The resulting loss is expressed as: in, The binary entropy loss function is expressed as: in, For the image Location label, For the device to view the image Predicted location; in, Represented as: in, and They are respectively from and Cut at the same position Image blocks, and They are respectively The mean and standard deviation, for and covariance, and Used to avoid the case where the denominator of a division is 0; in, Represented as: in, For the image Location label, For the device to view the image Predicted location.

Images